Presenters

Source

Unpacking the Argo CD Sync: A Deep Dive into the GitOps Engine 🚀

Hey tech enthusiasts! Ever wondered what really happens under the hood when Argo CD orchestrates your application deployments? Today, we’re pulling back the curtain on the intricate dance between Argo CD and its GitOps engine. Alexandre Gaudreault, Staff Software Engineer at Intuit and Argo maintainer, recently shed some light on this complex process, and we’re here to break it down for you! 💡

Intuit, a global fintech giant behind brands like TurboTax and Credit Karma, is a major player in the open-source community, even being instrumental in the creation and open-sourcing of Argo CD itself. They’re now working on the GitOps Promoter to further enhance continuous delivery workflows. But let’s dive into the heart of the matter: the Argo CD sync process.

The GitOps Engine: The Unsung Hero ⚙️

The GitOps engine is the core mechanism responsible for managing your Kubernetes clusters within Argo CD. It performs three crucial tasks:

  1. Cluster Caching 💾: It maintains an up-to-date cache of Kubernetes objects across all connected clusters in your Argo instance. This ensures quick access to the current state of your resources.
  2. Diffing 🧐: This is where the magic of comparison happens. The GitOps engine receives your desired application manifests and compares them against the live objects in your cluster. It then highlights the differences, informing you exactly what needs to change.
  3. Syncing 🔄: This is the action phase! The GitOps engine is directly responsible for applying your resources to the destination cluster, making your live environment match your desired state defined in Git.

However, the GitOps engine is just that – an engine. It needs a vehicle to drive, and that vehicle is Argo CD.

Argo CD: The Conductor of the Sync Orchestra 🎻

Argo CD is the tool that triggers the sync operation. You can initiate this through various means:

  • CLI: Using the argocd command-line interface.
  • UI: Clicking the sync button in the Argo CD web interface.
  • Auto Sync: Enabling automatic synchronization for your applications.

Regardless of the method, Argo CD acts by patching the Application resource. Kubernetes resources typically have spec and status. Argo CD introduces a third property: operation. When this operation field is set, Argo CD detects it and initiates the sync mechanism.

A Simple Sync in Action: Creating a ConfigMap ✨

Let’s walk through a basic scenario: creating a ConfigMap.

  1. Trigger: You click the sync button in Argo CD.
  2. Patching: Argo CD patches the Application resource to set the operation field.
  3. Comparison: The GitOps engine compares the desired ConfigMap state from Git with the live cluster state. It identifies that the ConfigMap is missing.
  4. GitOps Sync: The GitOps engine receives this information and knows it needs to create the ConfigMap.
  5. Application: It performs an action equivalent to kubectl apply.
  6. Success: The ConfigMap is successfully created, and your application is in sync.

While this might seem straightforward, the process is more nuanced than a simple kubectl apply.

The Inner Workings of a Sync: Beyond kubectl apply 🛠️

The GitOps engine orchestrates your sync with remarkable detail:

  • Resource Ordering 📜: Argo CD enforces a hardcoded order for resource creation. For example, it ensures ConfigMaps and Service Accounts are created before Deployments, as Deployments often reference these resources. This is managed by a preset list of Kubernetes resources with a defined order.
  • Task-Based Application 🎯: Your manifests aren’t applied all at once. The GitOps engine treats each resource as an individual task, applying them sequentially. This ensures a controlled and predictable deployment.
  • Dry Run First! 💨: Before any actual changes are made to your cluster, Argo CD performs a dry run for all resources. This crucial step catches YAML errors or potential issues, preventing partial failures during the apply phase.
  • Apply Step 🌟: If the dry run is successful, Argo CD proceeds to the apply step, sequentially creating or updating your resources in the determined order.
  • Customization Options 🔧: You can influence the apply process with options like replace (similar to kubectl replace) and force. These can be applied globally or as annotations on specific resources, giving you fine-grained control.

Navigating Complex Scenarios: Pruning and Renaming 🔄

What happens when you need to rename a resource? Kubernetes doesn’t allow direct renaming. Argo CD handles this through a pruning process:

  • Deletion and Recreation: Argo CD detects the rename, deletes the old ConfigMap, and then creates a new one with the desired name.
  • Impact of Pruning: While simple for a ConfigMap, imagine renaming an Ingress. Without careful handling, this could lead to an outage as traffic is temporarily misdirected.
  • pruneLast Option 💡: To mitigate such risks, you can use the pruneLast option. This applies the new resource first, then deletes the old one, minimizing downtime.
  • pruneConfirm for Safety ✋: For critical resources, the pruneConfirm option is invaluable. It pauses the sync before pruning, requiring manual confirmation in the UI or via the CLI. This gives you a critical safety net.

Sync Hooks: Extending the Sync Lifecycle 🪝

Sync hooks allow you to execute custom logic before, during, or after your sync operation. They are typically implemented as Kubernetes Jobs but can be any Kubernetes manifest.

  • Phases: Hooks can be defined for preSync, sync, and postSync phases. You can also define syncFail hooks to execute actions when a sync fails.
  • Execution Flow:
    1. Dry Run: All resources, including hooks, undergo a dry run.
    2. Pre-Sync Phase: Pre-sync hooks (like a schema migration job) are executed. By default, Argo CD deletes and recreates these jobs to ensure they run.
    3. Sync Phase: Your main application resources are applied.
    4. Post-Sync Phase: Post-sync hooks (like smoke tests) are executed.
    5. Health Checks: Argo CD monitors the health of your running tasks and hooks.
    6. Sync Fail Phase: If any part of the sync fails, the syncFail hooks are triggered. These hooks are also subject to deletion policies.
  • Deletion Policies: You can control when hooks are deleted:
    • hookDeletionPolicy (Default): Delete hooks before creation.
    • hookSucceed: Delete hooks after they successfully complete.
    • hookFail: Delete hooks after they fail (during the sync fail phase).

Sync Waves: Orchestrating Resource Dependencies 🌊

Sync waves provide a powerful way to order resources within a sync phase, especially useful for complex custom resources with dependencies.

  • Not for Application Orchestration: It’s important to note that sync waves are for ordering resources within an application, not for orchestrating syncs between different applications.
  • Wave Ordering: Resources are assigned a sync wave number. Lower numbers are processed first.
  • Health Dependency: To progress to the next sync wave, all resources in the previous wave must be healthy according to Argo CD’s health checks. This ensures a robust and sequential rollout.
  • Phase-Specific: Sync waves are applied after each phase (pre-sync, sync, post-sync), allowing for structured ordering within each stage of the sync.

Argo CD Specific Sync Features: Retries, Timeouts, Auto Sync, and Self-Heal 🛡️

Beyond the GitOps engine’s core functionalities, Argo CD offers several features to enhance the sync process:

  • Retry Policy 🔁: If a sync fails, Argo CD can automatically retry. You can configure the retry limit (including infinite retries with -1) and the retry interval.
  • Sync Timeout ⏳: To prevent syncs from running indefinitely, a global sync timeout can be set. This acts as a safety net, failing the sync after a specified duration, even if retries are configured.
  • Auto Sync 🤖: When enabled, auto sync triggers a sync automatically upon detecting a new commit in your Git repository. You can even restrict auto sync to specific file paths in a mono-repo. Crucially, auto sync only triggers once per commit and will eventually fail if retries are exhausted.
  • Self-Heal 🩹: This feature addresses unexpected Kubernetes changes that cause your application to go out of sync. Argo CD will retry the last successful sync to restore the desired state. Self-heal does not trigger for degraded applications; it only acts when the cluster state deviates from the desired state without a sync operation occurring. For self-heal to work effectively, your last successful sync must have been healthy.

Key Takeaways for Your Argo CD Journey 💡

As you navigate the complexities of Argo CD sync, keep these points in mind:

  • Clear Boundaries: Understand the distinct roles of the GitOps engine and Argo CD. When contributing, know where features belong.
  • Simplify Where Possible: Avoid over-complicating your sync process with unnecessary hooks or sync waves. Use them judiciously to make your deployments easier to understand.
  • Health is Paramount: The health status of your resources is critical. It influences pruning, sync waves, and the overall success of your sync.
  • Auto Sync Limits: Remember that auto sync has limitations and won’t retry indefinitely.
  • Self-Heal Context: Self-heal is a powerful tool for drift detection but relies on a healthy last successful sync and doesn’t address degraded application states.

Understanding these nuances empowers you to leverage Argo CD more effectively, ensuring smoother, more reliable application deployments. Happy syncing! ✨

Appendix