Presenters

Source

Taming Complexity: How SpotOn Revolutionized Engineering with Backstage 🚀

In the fast-paced world of tech, growth is often a double-edged sword. For SpotOn, a company powering mobile payments and management systems, rapid expansion through acquisitions brought incredible momentum but also a tangled web of systems and processes. Imagine inheriting not one, but three distinct startups, each with its own unique tech stack and deployment methods, all crammed into one organization! That was SpotOn’s reality. With a staggering 1,200 repositories and a constant barrage of “firefighting” from the operations team, clarity and focus felt like distant dreams.

Enter Backstage. Mike, a Staff Engineer, and Eli, an Engineering Manager at SpotOn, shared their transformative journey, revealing how Backstage became their secret weapon. The result? A newfound sense of clarity and focus, a meticulously organized catalog of 875 components, and the empowerment of a dedicated platform engineering team of 30 individuals ready to tackle scalability head-on.

Day One: Building a Foundation of Catalog Quality 🏗️

The initial setup of Backstage is surprisingly smooth, thanks to its excellent out-of-the-box features, comprehensive documentation, and a rich plugin ecosystem. However, the real work lies in organization and prioritization. The speakers’ golden rule for day one? Catalog quality is paramount.

Here’s how SpotOn prioritized building a robust catalog:

  • Embrace Software Templates, Ditch Mega Scripts: Instead of complex, one-off ingestion scripts, SpotOn champions the use of software templates. These templates are universally accessible, making it easy for everyone, from product managers to UI designers, to contribute. More importantly, they act as a powerful guide, defining what “high quality” looks like for various component types (services, websites, libraries) and enforcing essential standards. For instance, services now must have API documentation! SpotOn ingeniously uses two core templates: one for documenting existing components and another for creating new repositories pre-loaded with best practices.
  • Sustain Quality with Automation: Maintaining catalog quality over time is key. This is where automation shines:
    • CI Actions for Vigilance: Automated checks via GitHub Actions (or similar CI/CD tools) can diligently monitor catalog info files. If standards evolve, these actions will promptly alert developers, ensuring continuous compliance.
    • GUI Validation for Ease: Plugins like the open-source “pluggy entity validation” offer a user-friendly interface for easily verifying catalog info files. The “catalog unprocessed entities” plugin is a lifesaver, providing a dashboard to diagnose why certain entities might be missing from the catalog.
    • Gentle Nudges for Compliance: For repositories that slip out of sync, SpotOn employs a smart archival process. This automated weekly system gently encourages developers to update their documentation before they can proceed with crucial tasks like feature flags or bug fixes. It’s a brilliant way to maintain standards without creating unnecessary friction.
  • Mastering Tooling with Composability: The speakers advocate for a “templates all the way down” approach. This means breaking down larger software templates into smaller, composable sub-templates. This allows for independent versioning and publishing of standards, enabling subject matter experts to maintain them efficiently and reducing the barrier to contribution. The result? Core standards are never duplicated, freeing up teams to focus on building catalog quality.

Day Two: Scaling Value and Sparking Cultural Transformation ✨

Transforming Backstage from a helpful tool into an indispensable organizational asset requires a two-pronged approach:

  1. The Power of “Getting Your Hands Dirty”: Even with the best tooling, sometimes a little manual effort is required. SpotOn initially faced a plateau in reducing their undocumented repositories to zero, as developers naturally prioritized actively developed code. The game-changer? A significant organizational re-org that mandated domain assignments for every repository. Even with executive backing, this push demanded substantial manual work. They meticulously created a spreadsheet of all repositories, used “get blame” to identify contacts, and scripted the creation of catalog info files for over 600 repositories! While this manual effort was crucial, they emphasized that platform engineering is the key to making such interventions obsolete in the long run. It’s a delicate balance between enforcing standards and pragmatically getting things done.

  2. Backstage: A Tree Bearing Abundant Fruit 🌳: Once you cultivate a high-quality catalog, Backstage blossoms into a powerful source of truth, driving profound organizational changes. SpotOn masterfully leveraged their ownership data to:

    • Seamless AWS Integration: By integrating Backstage into their Terraform deployments, they dynamically inject domain ownership tags into AWS resources. This means every piece of infrastructure is automatically tagged with its responsible team.
    • Smart Kubernetes Labeling: For their poly-repo Helm deployments, a custom label controller ensures that pods, deployments, and services are tagged with domain metadata at runtime. This eliminated the tedious task of modifying hundreds of individual CI/CD pipeline definitions.

This outward flow of ownership information unlocked a cascade of benefits:

  • Automated Alert Routing: When an issue arises, alerts are now intelligently routed to the correct domain owners, eliminating the frustrating “software archaeology” of figuring out who’s responsible.
  • Empowered Cost Reporting: Teams can now accurately track the costs associated with their services and domains, fostering a culture of cost awareness and driving significant savings.
  • Just-in-Time Production Access: Developers on call can safely request production access, but only for the specific resources they own. This dramatically reduces the potential “blast radius” of any mistakes.

The ultimate lesson from SpotOn’s journey is clear: Build an exceptional, high-quality catalog, and then use that foundation to solve problems across your organization. Backstage, even for developers who might have only interacted with it once to populate the catalog, becomes an invisible yet indispensable engine powering critical operational and cultural transformations. It’s a true testament to the power of clarity and focus in the complex world of software development. 💡✨

Appendix