Presenters
Source
How Backstage Became the Unsung Hero of Our AppSec Program 🚀
Hey everyone! Today, we’re diving deep into a fascinating three-year journey that transformed RVO Health’s application security program. If you’ve ever felt the pain of scattered security data, unclear ownership, and the sheer effort it takes to get critical vulnerabilities fixed, then this story is for you. Our speaker, Chris, a Principal Engineer at RVO Health, shared how Backstage, specifically its software catalog, became the backbone of their entire appsec strategy.
Let’s break down how they achieved this, making complex security challenges manageable and even exciting!
The RVO Health Revolution: Merging Giants, Managing Chaos 🌐
RVO Health, though you might not know the name, is a significant player in the health and wellness space. Formed in 2022 from the merger of two established companies, they instantly brought together over 300 engineers, each with their own unique tech stacks and tools. Imagine the challenge: a sprawling infrastructure, over 1,000 GitHub repos, and hundreds of cloud accounts across different providers.
Early on, the need for a developer-friendly and proactive approach to security was clear. The reality? Spreadsheets, Slack messages, and a frustrating lack of clear ownership data. When a critical vulnerability surfaced, getting the right information to the right people was a Herculean task.
Enter Backstage: The Software Catalog as the Foundation 🛠️
While the security team was busy building the foundations of their appsec program, the central platform team turned to Backstage. Their initial focus? The software catalog to meticulously track ownership. They knew this information would be invaluable for security, and frankly, for their own sanity.
They chose a path less traveled, moving beyond traditional CMDBs. The investment in building out the software catalog was significant, even when teams initially questioned its value. By leveraging open-source entity providers and building some custom ones, they automated as much as possible. Gentle nudges via Slack and a small plugin to track progress became part of their daily workflow.
Key Takeaways:
- The Challenge: Managing diverse tech stacks and vast codebases post-merger.
- The Problem: Scattered security data, unclear ownership, and high friction for vulnerability remediation.
- The Solution’s Start: Backstage’s Software Catalog for tracking ownership.
- The Effort: Significant investment, automation, and iterative development over time.
- The Result: Around 90% adoption and a system that largely runs on autopilot! ✨
Team Pages: Bringing Context and Hierarchy to Security 👨💻
Inspired by Spotify’s approach to showing aggregated, user-relevant data, RVO Health adopted the concept of Team Pages. These are essentially hubs of key information specific to each engineering team. They discovered that rolling up data at a team or group level offered the best Return on Investment and fit seamlessly into the Backstage catalog ecosystem.
Crucially, Team Pages were designed with organizational hierarchy in mind. This means managers and leaders can easily view security overviews for entire departments or parent teams, making strategic security planning much more accessible.
Inside the Team Page:
- Security Overview Card: This is a vital feature, signaling exactly what security priorities a team should be focusing on.
- Powered by Security Metrics Plugins: These have become one of the most heavily utilized features of Backstage at RVO Health.
Bridging the Gap: Integrating Security Tools with Backstage 💡
RVO Health uses tools like Wiz for cloud account security and GitHub Advanced Security for code security. While these tools are excellent at their core functions, they can generate a ton of noise. They often lack immediate context around ownership and represent yet another place for development teams to check.
This is where Backstage truly shines. They leverage the strengths of these security tools but supplement where they can create unique value.
The Evolution of the Security Plugin:
- Early Days: Started as a “quick and dirty” table of Wiz issues, testing the waters for Backstage’s value. Initial usage was low, but some teams found it helpful.
- Expansion: Realizing the need for a broader view, they integrated GitHub findings, aiming to provide just enough information to drive engineers towards resolution without overwhelming them.
- Maturity: The plugin evolved to include quick overview cards and, more recently, a historical view. This historical perspective has been instrumental in helping teams visualize and demonstrate progress over time.
The Impact:
- Single Pane of Glass: Teams now have one central place to view their security posture for the assets they own and operate.
- Motivation: Seeing security metrics clearly displayed encourages teams to actively drive improvements (“numbers down and to the right”).
From Passive Data to Active Remediation: Slackbot to the Rescue 🤖
While the security plugin is powerful, Chris pointed out a key observation: teams don’t naturally wake up and check this data. To bridge this gap and drive action, they introduced a Slackbot.
Teams can invite the Slackbot to their channels and configure weekly notifications. This delivers a summary every Monday, complete with direct links for deeper dives and remediation. Major business units also have dedicated collaboration channels with security, where these notifications are sent, fostering a more proactive approach to addressing specific issues.
The Workflow:
- Weekly Summaries: Delivered via Slack, driving awareness and action.
- Collaboration Channels: Dedicated spaces for security discussions and progress tracking.
- Impactful Combination: The synergy of Backstage and Slack notifications has proven highly effective in driving team engagement.
Real-Time Alerts: Closing the Loop on Critical Vulnerabilities ⚡
The security team, fully on board by this stage, wanted to tighten the loop between detection and remediation, especially for critical and high-severity vulnerabilities. Their solution? Leveraging webhooks and the Backstage event system for real-time notifications.
Now, when a critical issue is detected, a team receives a Slack message almost immediately after the tooling identifies it. This is all powered by Backstage! Chris shared the incredible anecdote of pushing through dependency fixes to their Backstage instance in a matter of an hour – a testament to the system’s agility.
The Long Game: Incremental Progress and InnerSource Wins 🏆
Chris emphasized that this transformation didn’t happen overnight. It was a journey of incremental steps, with contributions from nearly a dozen people over a three-year period. The result? Teams now have a standard, familiar place to track their security efforts, and Backstage has played a crucial role in maturing their appsec program, reducing friction, and accelerating security fix times.
What’s truly inspiring is the inner-source contributions from the security team itself. They’re now using Backstage to power quarterly access campaigns – a complex, cross-organizational task built entirely within Backstage!
The Power of Building on Backstage:
- Simplicity: Plugins leverage core Backstage features, keeping development focused and manageable.
- Accessibility: “Everything you need to solve a problem is literally just a pull request away.”
- Ease of Iteration: Each subsequent feature becomes easier to build.
- Empowerment: Even individuals with limited software development experience are leveraging Backstage to ship value.
Chris concluded by giving a massive shout-out to the Backstage community, highlighting how it’s empowering teams to solve real problems in scalable, simple, and sustainable ways.
If any part of this journey resonated with you, don’t hesitate to reach out and chat! The power of Backstage lies in its ability to adapt and solve your unique challenges.