Presenters

Source

🚀 Stop Doing GitOps Wrong: 5 Argo CD Pitfalls and How to Fix Them

GitOps starts as a dream. You install Argo CD, deploy your first few applications, and the team feels unstoppable. But fast forward six months, and many organizations find themselves staring at a tangled mess that refuses to scale.

Regina Voloshin, an Argo CD maintainer at Octopus Deploy and a top contributor for the last 6 months, identifies the hidden traps that turn a clean setup into a maintenance nightmare. With only 3 women among 35 maintainers in the project, Regina brings a crucial perspective on how to build sustainable, developer-friendly delivery pipelines.

If your GitOps journey feels like it is hitting a wall, you are likely falling into one of these five common anti-patterns.

1. 🛑 The Hardcoding Trap: Helm and Kustomize Data

Argo CD provides three ways to hardcode Helm data directly into the Application Custom Resource (CR):

  1. source.helm.parameters
  2. source.helm.values
  3. source.helm.valuesObject

Regina’s advice is simple: Do not do this. 🙅‍♀️

The Challenge

This pattern mixes two different lifecycles. Developers own Helm values, which change constantly. Operations teams own the Argo CD Application manifests, which should remain static.

The Impact

  • Audit Nightmares: Security teams cannot easily find where settings live.
  • Broken Local Testing: If a developer wants to test a Helm chart locally, they cannot. The application becomes coupled to Argo CD features. To debug, a developer would have to run Argo CD locally just to generate a manifest.
  • CI Complexity: Your CI system now has to hunt through Argo CD manifests just to update a container image.

✨ The Fix: Store all Helm information in values.yaml files within Git and have the Argo CD application reference those files using the source.helm.valueFiles spec. This maintains a clear separation of concerns.

2. 🏗️ The Imperative Impulse: Dynamic Application Creation

Many teams transition to Argo CD by bringing along their old habits. They use a custom CLI or script to pull data from an external database, generate a manifest on the fly, and use kubectl apply or the Argo CD API to push it directly to the cluster.

The Tradeoff

While this feels “automated,” you lose the core benefit of GitOps: a declarative source of truth.

The Impact

  • No History: You lose the Git audit trail of what changed and why.
  • Disaster Recovery Failure: If your cluster vanishes, you cannot simply “re-sync” from Git because the source of truth lives in an imperative script or a separate database.

✨ The Fix: Use Git as your single source of truth. If you have an external system managing configurations, have that system commit the manifests to Git first. Let Argo CD observe the Git change and sync it to the cluster naturally. 🌐

3. 🥪 The “Helm Sandwich” Anti-Pattern

The Helm Sandwich occurs when you package Kubernetes manifests in a Helm chart, reference that chart in an Argo CD Application, and then package that Application manifest inside another Helm chart.

The Challenge

This creates two layers of Helm templating. It forces developers to understand the inner workings of Argo CD just to make a business logic change.

The Impact

  • Bad Self-Service: Developers become dependent on the Ops team to navigate the layers of templates.
  • Pointless Versioning: Teams often try to version the Argo CD Application chart. However, Argo CD Applications are static links between a repo and a cluster; they do not need versioning in the way business apps do.

✨ The Fix: Stop reinventing the wheel. Use ApplicationSets for advanced templating instead of wrapping everything in Helm. 🛠️

4. 📉 Writing Individual Applications vs. Application Sets

Imagine you have 5 infrastructure applications (like Prometheus or Cert-Manager) that you need to deploy across 10 Kubernetes clusters.

The Challenge

If you write individual manifests, you are now managing 50 separate Application objects. This is a recipe for configuration drift and manual error.

✨ The Fix: Use ApplicationSets. A single ApplicationSet can replace all 50 individual manifests. It allows you to categorize clusters and deploy across them using a single template. 🦾

5. ❓ Q&A: Flexibility and Complexity

During the session, the audience raised excellent points regarding the trade-offs of these “best practices.”

Q: Does moving to ApplicationSets reduce my flexibility? I like being able to tweak individual apps. A: (Answered by the session moderator) Flexibility should live in the manifest folders, not the Application CR. If you use a Git generator in an ApplicationSet, you can still have per-folder configurations (like ignoreDifferences). This keeps your source of truth clean and centralized. 🎯

Q: Is using multiple values files for a single application a bad idea? A: It is not inherently “wrong” because it allows for inheritance, but it is anti-pattern-ish. It increases the cognitive load on developers who have to figure out how three different overlays are merging to create the final value. It is usually better to simplify the structure using ApplicationSets to handle environment-specific overrides. 🧠

💡 The Path to “True” GitOps

To move from a “messy” setup to a scalable one, Regina suggests shifting your strategy from the Red (Anti-patterns) to the Green (Best Practices):

❌ The “Red” Way (Avoid) ✅ The “Green” Way (Adopt)
Hardcoding Helm/Kustomize data Referencing configuration files in Git
Imperative CLI/API creation Declarative manifests in Git
The “Helm Sandwich” Using ApplicationSets
Managing 50+ individual Apps Using ApplicationSets for multi-cluster

By following these principles, you ensure that your developers remain independent, your security team stays happy, and your Argo CD setup remains rock-solid as you scale. 🚀✨

You can find Regina Voloshin on LinkedIn or the CNCF Slack for more insights into Argo CD and Kubernetes delivery.💾📡

Appendix