Presenters

Source

From “App of Apps” Chaos to Scalable GitOps: A Kaltura Success Story 🚀

Navigating the complexities of managing a vast application landscape is a challenge many organizations face as they embrace GitOps. Kaltura, a growing company, found themselves at a critical juncture, managing over 2,000 production applications across dozens of cloud regions. Their initial approach, while effective for smaller deployments, was quickly becoming a significant bottleneck. Let’s dive into how they transformed their Argo CD operations from a monolithic headache to a streamlined, scalable powerhouse! 💡

The “App of Apps” Monolith: A Bottleneck in Disguise 🚧

The “App of Apps” pattern, where a single Argo CD application orchestrates others, initially seemed like a logical step for managing numerous applications across different repositories and environments. However, for Kaltura, this pattern evolved into a major operational hurdle.

  • Platform Team Overload: The central “App of Apps” manifest turned the platform team into a bottleneck. A constant stream of developer pull requests for changes led to delays and hindered the team’s ability to focus on innovation. 😫
  • Unintended Dependencies: Applications became tightly coupled, making changes complex and risky. A modification in one area could have unforeseen ripple effects across unrelated services.
  • Single Point of Failure: A single syntax error or bug in the main “App of Apps” manifest could halt deployments for everyone, creating a critical vulnerability.
  • Lack of Flexibility: The static nature of the “App of Apps” manifests prevented the implementation of dynamic logic or templating, limiting adaptability.
  • Performance Woes: As the number of applications grew, developers experienced frustratingly slow sync operations and basic management tasks within Argo CD.

The breaking point? The decision to expand to three additional cloud regions. The growing frustration with Argo CD performance and the “App of Apps” pattern became a recurring theme in daily stand-ups, signaling a clear need for a more robust solution.

Enter Argo CD Application Sets: The Scalability Game-Changer! ✨

Argo CD Application Sets emerged as the hero Kaltura desperately needed. This dedicated Custom Resource Definition (CRD) and controller are specifically engineered to tackle the limitations of the “App of Apps” monolith.

Why Application Sets Shine: 🌟

  • Unmatched Scalability: Application Sets are built for scale, effortlessly handling a massive number of applications with significantly less strain on the platform team.
  • Developer Empowerment & Self-Service: The reins of application deployment are handed over to developers, granting them the autonomy and self-service capabilities they crave. 👨‍💻
  • Native Auto-Discovery: Say goodbye to complex workarounds! Application Sets natively discover applications and environments, streamlining the management process.
  • Decoupled Architecture: Applications are decoupled, minimizing dependencies between teams and services, leading to more agile development.
  • Intelligent Logic & Templating: Application Sets unlock the power of complex logic and advanced templating, offering unparalleled flexibility.

Application Sets in Action: The Power of Generators 🛠️

The true magic of Application Sets lies in their generators. These define how applications are discovered and deployed, opening up a world of automation:

  • SCM Generator: This generator intelligently scans Git providers (GitHub, GitLab, Bitbucket) for repositories that match your defined rules. It automatically identifies deployment environments and targets specific clusters, making multi-environment deployments a breeze. 🌐
  • Pull Request Generator: Imagine ephemeral preview environments! This generator creates temporary environments for developers to test code against the latest Kubernetes resources. These environments are automatically cleaned up once the pull request is merged or closed, saving valuable resources. 👾
  • Cluster Generator: Leverage cluster labels to intelligently deploy applications to the right places. This allows for automatic deployments to newly provisioned clusters, ensuring consistency and speed.

The Migration Journey: A Smooth Transition 🛤️

Kaltura’s migration from the “App of Apps” pattern to Application Sets was a carefully orchestrated process:

  1. Deploy with Application Sets: Start by deploying your applications using the new Application Set CRDs.
  2. Application Set Takes Control: The Application Sets will automatically detect and assume management of your existing applications.
  3. Phased Removal: Gradually remove applications from the legacy “App of Apps” manifests.
  4. Decommission the Old: Once all applications are managed by Application Sets, the “App of Apps” can be safely decommissioned.

The Next Level: “App of App Sets” for Ultimate Control 👑

As Kaltura continued to scale, they encountered a new challenge: managing a vast number of Application Sets themselves. The solution? “App of App Sets.” This pattern leverages “App of Apps” to manage the Application Sets, maintaining a clear hierarchy and allowing the platform team to retain oversight while still benefiting from the power of Application Sets.

Pro-Tips for Your GitOps Journey! 💡

The Kaltura team shared some invaluable gems of wisdom gleaned from their experience:

  • Sync Policy is Key: Configure sync policies to precisely control how and when applications are synced, including the power of autosync and self-healing.
  • Argo CD Diff for Safety: Always use the Argo CD CLI to preview changes before applying them. This simple step can prevent accidental misconfigurations and save you from deployment headaches.
  • OIDC Authentication for Security: Implement OIDC for secure, credential-less authentication. This eliminates static credentials and helps mitigate rate limiting issues with Git providers. 🔐
  • Treat Argo CD Applications as Code: Apply the same rigorous code practices you use for your application code to your Argo CD application manifests. This means automatic PR validations and thorough code reviews.
  • Master the Art of Labels: Leverage labels extensively! They are crucial for Argo CD to understand where and how applications should be deployed, enabling powerful automation.

Tangible Wins: The Results Speak for Themselves! 🎉

The migration to Application Sets delivered remarkable improvements:

  • Reduced Operational Overhead: The platform team transitioned from reactive PR management to proactive innovation and strategic roadmap development.
  • Empowered Developers: A staggering 200 developers are now empowered to deliver features at an unprecedented pace, with 20 new features rolled out to the dev environment in just the last week! 🚀
  • Optimized Resource Usage: Significant reductions in CPU and memory consumption from the Argo CD controller were observed.
  • Near Real-Time Sync: Application reconcile latency plummeted, bringing synchronization capabilities closer to real-time.

Key Takeaways for Your Organization: 🎯

  • Patterns over Pure YAML: Focus on adopting scalable patterns rather than getting lost in individual YAML files.
  • Developer Ownership is Paramount: Empower your developers with ownership and foster a self-service culture.
  • Generators are a Force Multiplier: Combine different generators to unlock powerful automation capabilities.
  • Migrations are Organizational Change: Approach migrations as an opportunity to introduce new ways of thinking and boost overall velocity.
  • Prioritize Value Delivery: Implement Developer Experience (DevEx) features that demonstrably improve productivity and deliver tangible business value.

Kaltura’s journey is a testament to the power of embracing evolving patterns and tools to overcome scaling challenges. By moving beyond the monolithic “App of Apps” and adopting Argo CD Application Sets, they’ve not only streamlined their operations but also unleashed the full potential of their development teams. What patterns are you considering to elevate your GitOps game? ✨

Appendix