Presenters

Source

Supercharging Developer Workflows: How RBC is Revolutionizing Its Internal Developer Portal with Backstage and Temporal 🚀

Ever feel like your development processes are stuck in the past? You’re not alone! Large organizations, especially those at the forefront of AI and modern infrastructure like RBC, often face the challenge of streamlining complex, multi-step processes that can bring development to a grinding halt. But what if there was a way to inject agility and robustness into these workflows, empowering developers and slashing those frustrating delays?

RBC’s cloud team has been on a mission to do just that, and their innovative approach, leveraging Backstage and the powerful workflow engine Temporal, is a game-changer for internal developer portals (IDPs). Forget slow, manual approvals and fragile automations – this is about building a future-proof development ecosystem.

The Bottlenecks: Why Backstage Alone Wasn’t Enough 🚧

As a major financial institution deeply invested in cutting-edge technologies, RBC encountered several hurdles with the standard Backstage scaffolder:

  • The Agony of Manual Approvals: Imagine processes that take days to get signed off by various stakeholders. This was the reality for RBC, with multi-step manual approvals creating significant drag on development cycles.
  • Fragile Automations, Endless Restarts: Existing automated workflows were notorious for failing. Without robust error handling, any hiccup meant starting from zero, a monumental waste of precious time and resources.
  • YAML’s Growing Pains: As the complexity of their automations grew, managing it all within YAML configurations became unwieldy and difficult to scale.
  • Secrets Management Jitters: For a large organization, centrally managing secrets within Backstage sparked valid security concerns.
  • The TypeScript Barrier: Extending Backstage scaffolder functionality often required TypeScript expertise, which wasn’t always readily available across their diverse platform engineering teams.

The Breakthrough: An Event-Driven Powerhouse for Agility ✨

RBC’s solution? They’ve strategically integrated Backstage with an external, event-driven workflow engine, and Temporal has emerged as their champion. This architectural pivot unlocks a host of incredible benefits:

  • Workflows That Breathe (and Wait!): The new system seamlessly handles long-living workflows that can pause for days, allowing for crucial human approvals from DBAs, customer experience managers, and more.
  • Resilience Built-In: Say goodbye to starting from scratch! Temporal’s built-in retry, rollback, and cancellation capabilities mean exceptions are handled gracefully, drastically improving system resilience.
  • Decentralized Smarts: Workflow execution and secrets management are now moved closer to where they’re needed, boosting both security and operational efficiency.
  • Polyglot Power for All: Developers can now build automations in their preferred languages – Java, Python, Go, you name it! This fosters greater autonomy and accelerates innovation.
  • Unified Visibility: The workflow engine provides a centralized UI for monitoring workflow progress, offering a clear, unified view for both users and systems, making troubleshooting a breeze.

Under the Hood: The Technical Symphony 🛠️

How does this magic happen? RBC’s enhanced architecture keeps Backstage’s familiar UI intact, using React JSON Schema Form for dynamic UI generation based on YAML templates. But here’s where it gets exciting:

  • Golang for Action: While user input remains driven by React JSON Schema Form, Golang now defines action configurations. This allows for sophisticated parameter passing, data transformations, and calls to native functions.
  • Delegating the Heavy Lifting: Backstage no longer bears the burden of workflow state management. This responsibility is handed over to the independent workflow engine, Temporal, which excels at event-driven patterns like the saga pattern.
  • Distributed Workforce: Specialized workers, coded in various languages, communicate with Temporal to execute specific tasks. These workers are responsible for interacting with cloud providers (AWS, Azure) and managing secrets locally.

Choosing Your Workflow Champion: Key Considerations 🏆

When selecting a workflow engine, RBC had a clear set of priorities:

  • Endurance: Long-running capabilities with the flexibility for human intervention.
  • Usability: An easy-to-use administrative UI for effortless troubleshooting.
  • Code-Centric: Workflows defined as code for version control and collaboration.
  • Flexibility: Portability across different cloud platforms.
  • Scalability & Cost: Efficient performance with reasonable licensing.

While options like Dapr Workflow and Argo Workflow were on the table, and cloud-specific solutions like Azure Durable Functions and AWS Step Functions offered similar features, Temporal stood out. Its open-source nature, multi-cloud hosting capabilities, Kubernetes-native deployment, low infrastructure demands, impressive real-world use cases, and extensive polyglot SDK support made it the clear winner.

The Demo Unveiled: Provisioning a Database in Minutes! 💻

The live demonstration was nothing short of impressive. A simplified database provisioning workflow showcased the power of this integrated system:

  1. Initiation in Backstage: A developer starts a “new database” workflow from a custom “Workflows” view within Backstage.
  2. Intuitive Data Gathering: Essential details like database name, environment, and storage quota are collected via a dynamic form powered by React JSON Schema Form.
  3. Smart Cost Estimation: Golang steps in to calculate estimated monthly cloud costs based on the provided storage requirements, allowing developers to iterate and refine.
  4. The Human Touch: The workflow gracefully pauses for a DBA’s manual approval, showcasing the critical human-in-the-loop capability.
  5. Temporal Takes the Helm: Once approved, Temporal orchestrates the entire “create database” workflow.
  6. Real-World Error Handling: A simulated connection error (a wrong password!) caused a Python worker to fail. However, Temporal’s robust retry mechanism kicked in. After the password was corrected in the worker, the workflow resumed without any manual intervention in Backstage.
  7. Success! Database Provisioned: The corrected worker successfully provisioned the database, verified live in PG Admin.
  8. The Future is Connected: The session hinted at even more exciting possibilities, like seamlessly integrating new databases with existing applications by selecting application types (Java, Python, Node.js) and triggering further automated deployments.

The Secret Sauce: A Thin Adapter Layer 🌐

A crucial detail is that Backstage doesn’t talk directly to Temporal. Instead, a thin adapter layer acts as the intermediary, providing REST APIs to Backstage. This layer handles the presentation of forms and manages submission and status updates, creating a clean separation that allows both systems to evolve independently.

Q&A Highlights: Digging Deeper 💬

The audience had insightful questions, and RBC’s team provided valuable clarity:

  • Production Power: In production environments, RBC leverages Terraform for infrastructure as code, with Terraform Enterprise handling deployments to cloud providers. The demo simplified this for illustrative purposes.
  • Approvals & GitOps Harmony: Approvals are seamlessly integrated with existing systems like ServiceNow or Jira, with Temporal orchestrating the overall process.
  • Scaffolder vs. Workflow Engine: Backstage’s scaffolder remains the go-to for simpler, one-off application repository deployments. The workflow engine is reserved for more intricate, multi-step, and potentially error-prone integrations.
  • Secure Secrets Management: Secrets are managed according to cloud provider best practices (e.g., Azure Key Vault for AKS) and are injected by the worker into the appropriate secret key store.
  • Why Temporal? (Again!): The open-source nature, multi-cloud support, Kubernetes deployment ease, low infrastructure overhead, impressive scalability, and polyglot SDKs were key decision drivers.
  • Multi-Tenancy Ready: Temporal’s robust features for namespacing and custom authentication ensure tenant-specific views and access control are achievable.
  • Community & Collaboration: With a developer community in the thousands, RBC envisions an “inner source” model where platform teams contribute their workflows to Temporal, making them readily consumable by application teams.

RBC’s innovative integration of Backstage and Temporal is a powerful testament to how forward-thinking organizations can overcome complex challenges, empower their developers, and build truly robust and agile internal development platforms. This is more than just an upgrade; it’s a revolution in how we build and deploy software! 💡🦾

Appendix