Presenters

Source

Say Goodbye to Staging Nightmares: How SeatGeek Built FE for Flawless Full-Stack Ephemeral Environments 🚀

Ever felt like you’re in a digital “video game battle” just to get your code onto staging? You’re not alone. For many engineering teams, shared staging environments, while a seemingly good idea, quickly devolve into a chaotic mess. Faulty deployments from one team can break everything for everyone else, leading to frustrating contention, unreliable test results, and a general lack of confidence in pushing even minor changes.

Zach Hammer and Colin Odell from SeatGeek know this pain all too well. They recently shared their journey in building “FE,” a revolutionary tool designed to orchestrate full-stack ephemeral environments using the power of Argo CD and other CNCF-native technologies. Their goal? To liberate developers from the shackles of unstable shared staging and unlock true productivity.

The Staging Stranglehold: Why Shared Environments Fail 😩

Let’s face it, shared staging environments are a breeding ground for problems as organizations scale:

  • Constant Contention: Developers spend more time fighting fires caused by others’ deployments than actually building features.
  • Degraded Health: The environment becomes so unreliable that test results are questionable, leading to risky production releases.
  • Skipped Testing: Frustration mounts, and crucial testing phases are often bypassed, injecting significant risk into the pipeline.
  • Eroded Confidence: Engineers become hesitant to deploy, even critical fixes, because they can’t be sure their complex, multi-system interactions will work.

Beyond Basic Review Apps: The Need for True Isolation 💡

Traditional “review apps,” which are great for testing multiple versions of a single application, simply don’t cut it when you’re dealing with complex, interconnected systems. They struggle with:

  • Cross-System Testing: It’s incredibly difficult to test how changes in one service impact another.
  • Asynchronous Systems: Isolating event-driven architectures like RabbitMQ or Kafka becomes a Herculean task.
  • Frontend/Backend Coordination: Managing changes across disparate frontend and backend repositories is a headache.

Enter FE: Your Personal Full-Stack Playground 🛠️

FE is SeatGeek’s answer to these challenges. It’s built to empower developers with isolated, full-stack ephemeral environments that mimic production without the chaos. The core principles behind FE are:

  • Semi-Isolated Application Constellations: Deploy groups of applications that can talk to each other, while still optionally leveraging stable versions of other services from the main staging.
  • Unwavering Staging Stability: Keep the main staging environment pristine and unaffected by the numerous temporary environments.
  • Ephemeral Resource Power: Provision temporary instances of databases (like PostgreSQL) or message brokers (like RabbitMQ) as needed for your environment.
  • Developer Delight: Offer an intuitive, easy-to-use experience that hides the underlying complexity.
  • Future-Proof Design: Build a system that can adapt and grow with evolving needs.

The Magic Behind FE: Argo CD and a Custom Operator ✨

The SeatGeek team initially dabbled with Argo CD’s “app of apps” pattern, but found it cumbersome for Helm chart development and difficult for developers to grasp. This led them to a more elegant solution: a custom operator built with the Rebar SDK.

This custom operator is the heart of FE. Developers now define their ephemeral environment using a simple YAML manifest, specifying:

  • Applications: Which apps to deploy, and crucially, from which branches or merge requests.
  • User Information: To track deployments, send notifications, and even facilitate cost allocation.
  • Additional Resources: Any specific databases or message queues required for the environment.

The operator then takes this manifest and orchestrates the entire deployment process, reporting the aggregated status back to the user.

The FE User Experience: Effortless and Empowering 👨‍💻

FE is designed with the developer at its core, making the process remarkably smooth:

  • Simple CLI Commands: Kick off environment management with a straightforward let's go FE manage command.
  • Interactive Environment Creation: A guided flow lets you easily select the applications, branches, and merge requests you need for your isolated environment.
  • Instant Gratification: As soon as you submit your request, you get immediate access to application URLs, health status, and expiration times. Notifications land in your Slack via their Mailroom framework.
  • Deep Dive Monitoring: Integrated links to DataDog dashboards provide unparalleled visibility into traces, logs, and the behavior of your ephemeral environment.
  • Automated Housekeeping: Environments automatically clean themselves up when merge requests are closed or their designated lifespan ends, with easy options to extend or retire them.
  • Mobile-First Testing: Support for full-stack testing extends to mobile applications, allowing them to seamlessly connect to your ephemeral endpoints.

The Impact: A Revolution in Developer Productivity 🚀

The adoption of FE at SeatGeek has been nothing short of phenomenal.

  • Over 90% of engineers actively use FE on a monthly basis.
  • 200 FE environments are created per week, showcasing the immense demand.

This shift has dramatically boosted developer confidence, simplified the testing of intricate, multi-system changes, and ultimately, fostered a more efficient and robust development lifecycle. If you’re tired of staging chaos, FE offers a compelling vision for the future of full-stack development.

Appendix