Presenters

Source

Say Goodbye to GitOps Guesswork: Preview Your Pull Request Changes in Seconds! 🚀

Ever stared at a pull request filled with Kubernetes YAML, Helm charts, or Kustomize patches and felt a cold dread? You’re not alone. As a developer or reviewer, you often hope those changes work, because mentally parsing templates to predict the final cluster state is, quite frankly, impossible. This “approve and pray” approach is a silent killer of confidence in GitOps.

But what if you could look into the future? What if you could see the exact impact of your changes before they even hit your cluster? Dag Bjerre Andersen from Eggmmont and Sergey Shevchenko from Tanga are here to tell us that future is now, thanks to the incredible Argo CD Diff Preview tool.

💡 The GitOps Review Dilemma: A Leap of Faith

Imagine you’re reviewing a PR: a few lines bumped in a version, a small tweak to a ConfigMap, or an update to a Helm chart. You see the templates, but you don’t see the final rendered manifests that Argo CD will actually apply. This creates a massive blind spot.

Dag highlights a crucial problem: you often approve changes hoping they turn out as expected. We need the equivalent of Terraform’s terraform plan for Argo CD – a crystal-clear preview of what’s about to happen.

🎯 The Golden Rules for Trustworthy Previews

When choosing a preview tool for your critical infrastructure, Dag emphasizes two non-negotiable principles:

  1. Reliability and Correctness Above All Else: A preview that’s only correct 95% of the time isn’t good enough. You need 100% accuracy, every single time. Otherwise, you won’t trust it, and it loses its value.
  2. Compare Your PR Branch to Your Main Branch (Not Your Live Cluster!): This is a game-changer. Comparing a static configuration (your PR branch) to a dynamic configuration (your live cluster, which constantly changes due to reconciliations, resource creation/deletion, and drifts) leads to non-deterministic output. You’ll see noise from cluster-sync issues or temporary states, not just your changes. Comparing PR to main ensures a deterministic and reliable output every run.

✨ Enter Argo CD Diff Preview: Your GitOps Oracle

The Argo CD Diff Preview tool directly addresses these challenges. It provides a beautiful, line-by-line comment on your pull request, showing precisely which applications and their underlying resources will update. This means you can:

  • Review code with confidence: No more surprises when merging to production.
  • Verify correctness: See exactly what Argo CD will render.
  • Gain visibility: Understand the full impact of changes to Helm charts, ConfigMaps, or deployments.

🛠️ How It Works: Two Powerful Modes

This agnostic tool simply takes two Git branches and generates markdown or HTML output. It offers two primary modes:

1. ☁️ Ephemeral Cluster Mode: Isolation and Simplicity

This mode is perfect for getting started quickly and ensures complete isolation:

  • The Flow: A developer makes a change, triggering an automated pipeline. The pipeline creates a local, ephemeral Kubernetes cluster (only accessible within the pipeline). It then installs Argo CD, applies applications from both the main and PR branches, lets Argo CD do its rendering magic, extracts the rendered manifests, compares them, and posts the diff to your PR.
  • Key Benefit: It uses a real Argo CD instance for rendering, matching your production configuration one-to-one. This guarantees reliability. You can run as many instances as you want in parallel.
  • Trade-off: Speed. Creating a cluster and installing Argo CD takes time. Processing 900 applications could take 80-90 seconds. For smaller changes (1-10 applications), expect around a minute, with about 50 seconds dedicated to startup time.
  • Setup: Dag highlights its ease of use – often just a copy-paste of a pipeline example and a few credentials.

2. ⚡ Preconfigured Cluster Mode: Blazing Fast Feedback

For those who crave speed, this mode connects to an already running Argo CD instance:

  • The Goal: Eliminate the 50 seconds of startup time from the ephemeral mode.
  • Crucial Warning: Do not connect this to your production Argo CD instance! The preview process involves creating and deleting applications, which would overload and crash your production system and create a terrible user experience.
  • The Solution: Use a dedicated Argo CD instance specifically for previews. If strict security requirements make exposing a cluster to pipelines difficult, self-hosted runners can manage security aspects like service accounts, RBAC, and network policies.
  • Benefit: Dramatically faster feedback loops, pushing processing times down to seconds.

📈 The Need for Speed: Optimizing Performance

The Eggmmont team has relentlessly focused on speed. Processing 900 applications in one go has sped up significantly, now taking only 80-90 seconds. For typical changes (1-10 applications), you’re looking at about a minute.

🏢 Egmont’s Journey: Sub-5 Second Previews for 1000 Applications!

Dag shares Eggmmont’s impressive setup:

  • They started with the ephemeral cluster setup, which worked well.
  • To achieve extreme speed without the cost of a dedicated cluster, they installed a second, slimmed-down Argo CD instance directly on their production cluster, alongside their main Argo CD.
  • This “lockdown mode” ensures the preview instance is namespace-scoped, with removed cluster roles and unnecessary components.
  • Combined with self-hosted runners, this setup processes 1000 applications in just around 5 seconds, providing incredibly fast and accurate feedback.

🏦 Tanga’s Advanced Playbook: Handling Thousands of Applications

Sergey Shevchenko from Tanga, an early adopter, manages a massive infrastructure:

  • 30 clusters across different cloud providers.
  • 2000 applications.
  • 30 engineers working on infrastructure.
  • They use a dedicated Argo CD instance solely for previews, within a cluster managing other Argo CD instances for production, staging, and infrastructure.

Tanga faced challenges with complex templating (Helm charts creating other applications) and slow CI pipelines (up to half an hour initially for rendering 2000 applications). They tackled this with:

  • Application Selection: Tools like manifestGeneratePaths and custom watch patterns (regular expressions) help the tool identify only the relevant applications to render.
  • Impact: Using watch patterns reduced their pipeline time from 30 minutes to 3 minutes.
  • Plugins: They also leverage custom Helm binaries and Vault integration to validate secrets during the preview.
  • Preconfigured Mode: Switching to a preconfigured Argo CD instance with self-hosted GitLab runners further slashed their pipeline time to just 30 seconds – a six-fold speed increase!
  • Result: Tanga now renders 48 applications in just 8 seconds, enabling their teams to merge rapidly without breaking things.

🚀 Your Next Steps: Start Simple, Go Fast!

Dag and Sergey offer clear advice:

  • Use the Real Deal: Always use a real Argo CD instance for rendering to ensure reliability and correctness.
  • Compare Branches Only: Stick to comparing your PR branch against your main branch for deterministic output.
  • Start Simple: Don’t get overwhelmed by advanced setups. Begin with the ephemeral cluster mode, see the value, and then optimize for speed if needed.
  • Be Creative: The tool takes branches and outputs text; you can likely integrate it into any workflow.
  • Ephemeral vs. Preconfigured: Both are valid. Ephemeral offers simplicity and isolation out-of-the-box. Preconfigured delivers speed but requires more setup effort.

Once you experience the safety and confidence of seeing every change before it hits production, you can never go back! This kind of feedback loop transforms your GitOps workflow from a guessing game into a precise, predictable operation.

Thank you for your time! If you have any questions, connect with Dag Bjerre Andersen or Sergey Shevchenko on LinkedIn or Slack!

Appendix