Presenters

Source

Unlocking the Developer Catalog: N26’s Journey to Empower Engineers 🚀

Ever felt the frustration of a powerful internal developer platform that’s just a little too internal? You know, the one with all the data, but accessing it feels like a scavenger hunt? At N26, a leading mobile bank, they faced this exact challenge with their Backstage catalog. Sebastian Poxhofer, a Tech Lead at N26, shared their brilliant journey to transform their catalog from a treasure trove into an accessible, actionable powerhouse for their engineering teams. Get ready to dive into how they unlocked their catalog, empowering engineers with self-service and automation!

The Foundation: N26’s Custom Catalog Core 🏗️

N26, founded in 2013, has grown into a significant tech enterprise. With roughly 1,600 employees, including about 800 engineers, managing around 1,200 repositories, the scale of their operations demands robust internal tooling.

Instead of the standard catalog-info.yaml, N26 utilizes a custom s-profile.json – a solution that actually predates Backstage itself! Their custom Backstage implementation reads these s-profile.json files, generating internal components, resources, and systems.

At the heart of their philosophy lies a crucial principle: off-hand for the user. Engineers don’t need to set annotations themselves unless they specifically want to override standard behavior. This approach is vital at scale; otherwise, you risk inconsistent data and the endless task of chasing down engineers to update their metadata.

This comprehensive catalog now boasts approximately 80,000 entities and captures the complete organizational structure, including teams, groups, components, and resources. To achieve this, N26 employs a large number of custom catalog processors that pull data from across the company – think GitHub, GitHub Actions, Workday, and Google. They even deduplicate metadata like user IDs from different systems, enabling seamless cross-referencing and data correlation.

The Access Puzzle: Why Engineers Needed More 🧩

Despite having a rich, comprehensive catalog, N26 encountered a significant hurdle: access. The internal portal team quickly became a bottleneck for integration requests, hindering agility. Engineers faced several pain points:

  • Local Testing Woes: Testing scripts against Backstage was a nightmare. Engineers resorted to manually extracting short-lived authorization tokens from browser developer tools, which was unreliable and time-consuming. These tokens often expired before scripts finished, leading to frustration.
  • Ad-Hoc Reporting Headaches: Principal engineers struggled with ad-hoc reporting, particularly for generating security reports, due to the difficulty in programmatic access.
  • Poor User Experience: The reliance on Chrome DevTools for token extraction was a clunky, inefficient user experience, and it primarily worked only for catalog data, limiting its utility for other API endpoints.

Paving the Way: Early Wins & Dynamic Credentials 🔑

N26 tackled these challenges systematically, starting with quick wins before scaling up their authentication mechanisms.

  • Quick Wins:

    • They enabled the “try out” feature within Backstage’s API docs plugin, making it easier for users to test OpenAPI endpoints.
    • They introduced a web UI functionality to fetch short-lived API tokens, a small but impactful improvement over manual extraction.

  • Scaling with Dynamic Credentials:

    • The team integrated JVKs (JSON Web Key Set) external service-to-service authentication into Backstage’s auth backend. This standard OIDC (OpenID Connect) backend functionality allows Backstage to validate JSON Web Tokens against an endpoint on their IDP (Keycloak).
    • Initially, this setup provided read-only access because JVKs, in this configuration, create Backstage service principles. This bypasses the typical user-specific permission system, meaning Backstage doesn’t directly know the individual user. Sebastian noted that Backstage’s experimental auth is on the horizon to address this, enabling full user principle resolution.

This shift to dynamic credentials had an immediate impact:

  • The CI team integrated GitHub Actions, leveraging its OIDC functionality for automatic authentication against Backstage. This enabled automated generation and validation of critical files like code owners within N26.
  • It also spurred the creation of CLI tooling for tasks like migration and hydration.

The Game Changer: Building the N26 CLI 🚀

Recognizing the need for a more robust, user-friendly solution, N26 drew inspiration from the GitHub CLI’s gh api command and decided to build their own powerful CLI.

Their goals were clear:

  • Automate IDP Operations: The CLI should handle all identity provider interactions seamlessly.
  • Secure Token Storage: Utilize the Mac OS keychain to securely save tokens and secrets.
  • Ease of Installation: Make it simple for engineers to get started.
  • Measure Adoption: Track usage with OpenTelemetry.

The CLI’s authentication flow is a masterclass in user experience:

  1. It reuses the JVKs workflow but intelligently switched the IDP to Device Auth. This is a game-changer, as it means engineers no longer face browser pop-ups for every new OIDC flow when running scripts.
  2. The initial setup involves a one-time device validation against the IDP.
  3. Once validated, the CLI obtains a refresh token and securely saves it to the keychain.
  4. On subsequent runs, the CLI loads the refresh token, checks its validity (TTL – Time To Live), and if expired or a 403 error is received, it automatically refreshes the token.
  5. With a fresh access token, the CLI then accesses the Backstage API via the JVKs endpoint, allowing Backstage to validate the token with the IDP and return the response.

To track the impact and adoption of this new tool, N26 implemented OpenTelemetry API for Go within the CLI. This data is sent to Statsig, an A/B testing tool that offers an OpenTelemetry endpoint, providing valuable insights into command usage and overall adoption patterns.

CLI in Action: Empowering Engineers with Automation 🛠️

The N26 CLI has truly become the go-to tool for engineers, fostering a culture of self-service and automation.

  • Generic API Access: The foundational PCLI API <path> command allows engineers to interact with any Backstage API endpoint directly from their terminal.

  • Key Subcommands & Use Cases:

    • PCLI catalog owns: This powerful command returns all entities owned by the current user. It works by first checking with the IDP (OIDC whoami endpoint) to get user credentials and metadata, including the user’s Backstage ID. It then hits the Backstage API to resolve group memberships and ultimately determine ownership. The output is flexible, providing a human-readable table by default, but also JSON or Backstage IDs for easy scripting.
    • PCLI catalog at: This command intelligently autodetects the context from the current Git repository and fetches the corresponding Backstage entity. It’s incredibly useful for mass changes across the codebase, especially when used with tools like multikter, and includes monorepo support.
    • PCLI catalog owner resolve-groups: Instead of just returning groups, this command resolves them to a deduplicated list of individual users, making it easier to identify direct contacts for specific code pieces.
    • Tech Processing: Commands like PCLI docs lint and doc serve inject custom logic for Backstage TechDocs processing.
    • Automated Code Ownership: The CLI handles the regular generation of code owner files on GitHub, all based on the s-profile.json data.
    • Advanced Automations: PCLI automations run target allows engineers to run specific scripts in the context of owned projects. It checks out repositories into a temporary folder and executes the script against their content. This is invaluable for principal engineers conducting validations or running complex analyses.
    • PCLI search: Provides a simple way to search for specific services or service types directly from the local environment.

The possibilities are endless, limited only by an engineer’s imagination. Users have scripted everything from sending automated emails to specific users to running complex validation tasks.

While the N26 CLI is not open source, its impact is far-reaching. It serves as a foundational component in their internal cloud code marketplace, providing agent skills for coding agents, particularly for identifying ownership and other catalog-related queries.

Empowering the Future ✨

N26’s journey to unlock their developer catalog is a testament to the power of thoughtful tooling and a user-centric approach. By systematically addressing access challenges, building dynamic authentication, and creating a powerful, intuitive CLI, they have transformed their Backstage catalog into a vibrant, self-service platform. This not only streamlines operations but truly empowers engineers to innovate and automate, leaving more time for what matters most: building incredible products.

Appendix