Presenters

Source

Beyond the Chart: Mastering Kubernetes Deployments with Rendered Manifests and OCI 🚀

Ever feel like you’re playing a guessing game with your Kubernetes deployments? You tweak a Helm chart or a Customize overlay, hit deploy, and… poof… something unexpected happens. If this sounds familiar, you’re not alone! Christian Hernandez, a Technical Marketing Engineer and Tech Lead at Cisco, recently shed some serious light on this common pain point at a recent tech conference, unveiling a powerful approach that brings clarity and control back to your GitOps workflows. Let’s dive into the world of rendered manifests and the incredible potential of the Open Container Initiative (OCI).

The Abstraction Trap: When Tools Obscure the Truth 🕵️‍♀️

We all love tools like Helm and Customize for their ability to tame YAML repetition. They’re fantastic for DRYing up your manifests and making them more manageable. However, Hernandez pointed out a significant drawback: these tools introduce an abstraction layer. This layer, while convenient, can often lead to a frustrating lack of visibility.

  • The Opaque Box: You’re no longer looking directly at what will be applied to your cluster. Instead, you’re interacting with a tool that generates that YAML.
  • Hidden Surprises: A seemingly minor version bump in an umbrella Helm chart, for example, can cascade into a wave of significant, and often opaque, changes. This makes tracking, auditing, and even collaborative development a real challenge.

The Rendered Manifest Pattern: Bringing Clarity to the CI Pipeline 💡

The solution? Shift the rendering process out of your GitOps tool and into your Continuous Integration (CI) pipeline. This is the essence of the rendered manifest pattern.

Here’s how it revolutionizes your workflow:

  • CI as the Renderer: Your CI engine, whether it’s GitHub Actions, GitLab CI, or another favorite, becomes responsible for executing helm template or customize build commands.
  • Immutable Artifacts: The output of this rendering is your raw, unadulterated YAML. This is then stored as an immutable artifact. Think of it as a snapshot of exactly what will be deployed.
  • The Deployment Branch: A dedicated “deployment branch” often serves as the home for these rendered manifests, treating them as deployable units.

The Game-Changing Benefits:

  • Crystal Clear Visibility: You see exactly what’s going to hit your cluster. No more guesswork! ✨
  • Reduced Deployment Risk: Any rendering bugs or unintended changes are caught during the CI phase, before they impact your live environment. This significantly reduces last-minute firefighting. 🚒
  • Leveraging Git’s Might: You can now apply robust Git workflows, like branch protection rules and detailed commit histories, directly to your deployment artifacts. 🌳

A Quick Note on “Environment Branches”: Hernandez clarified that these branches aren’t for traditional Git Flow merges. They act more like organized containers for your rendered manifests, maintained by automation. They’re essentially release bundles, not channels for manual promotion.

OCI: The Universal Artifact Standard 📦

The conversation then took an exciting turn towards the Open Container Initiative (OCI). While initially conceived for container images and runtimes, OCI has evolved into a powerful, generic standard for storing and distributing any type of artifact.

  • Beyond Containers: OCI registries are designed to store “blobs” of arbitrary data. This means they can house not only container images but also:
    • Helm Charts 🚀
    • Software Bill of Materials (SBOMs) 🧾
    • Policy Bundles 🛡️
    • And, crucially for our discussion, YAML manifests! 📝
  • ORAS: Your OCI Toolkit: The ORAS (OCI Registry as Storage) project is key here. It empowers you to treat your OCI-compliant registry as a universal storage solution for all your artifacts.
  • Argo CD’s OCI Embrace: The real magic happens with Argo CD’s growing support for OCI (starting from v3.1). This allows Argo CD to directly pull and deploy YAML manifests stored within OCI registries.

The Ultimate Synergy: Rendered Manifests + OCI + Argo CD = GitOps Nirvana 🌌

When you combine the rendered manifest pattern with OCI artifact storage and Argo CD’s OCI support, you unlock a truly next-generation GitOps experience.

The New, Powerful Workflow:

Instead of rendering your manifests to a Git branch, you render them directly into an OCI repository. Argo CD then pulls these directly from the OCI registry for deployment.

This creates an artifact-based GitOps workflow, treating your deployments with the same rigor you apply to your compiled application artifacts.

Key Advantages of this Combined Approach:

  • Workflow Separation: A clean, distinct separation between your Git development workflow and your deployment workflow. แยก
  • Reduced Dependencies: You minimize your reliance on Git and Helm for the actual deployment process.
  • Enabling New Frontiers: This is a game-changer for:
    • Edge Clusters: Where network connectivity might be intermittent. 📡
    • Disconnected/Air-Gapped Environments: Crucial for highly secure or isolated setups. 🔒
    • Regulated Scenarios: Where direct Git access to production environments is restricted. ⚖️

Navigating the Challenges and Tradeoffs 🤔

While this pattern offers immense power, it’s not without its considerations:

  • Diffing Dilemma: The primary challenge is diffing rendered manifests when they’re stored as opaque blobs in OCI registries. You’ll need to place a high degree of trust in your CI engine’s rendering process.
  • Loss of Familiar Benefits: By moving away from direct Git/Helm interaction for deployment, you might miss out on some of the inherent benefits these tools provide.
  • Workflow Complexity: This approach requires careful planning and management of your CI and artifact workflows.
  • Admin vs. Developer Focus: This pattern might resonate more strongly with operations and administrator teams who value explicit control and auditability, potentially requiring a shift in mindset for developer-centric teams.
  • Sealed Secrets Caveat: Storing plain text secrets in rendered manifests is generally not recommended, especially if you’re using tools like Sealed Secrets. Hernandez suggests the External Secrets Operator as a more robust alternative for managing secrets in this context.

The Future of GitOps is Here! ✨

Christian Hernandez’s insights paint a compelling picture of how we can evolve our GitOps practices. By embracing the rendered manifest pattern and leveraging the universal capabilities of OCI, we can build more robust, transparent, and flexible deployment pipelines for Kubernetes. This is an exciting evolution that promises to bring greater control and confidence to our deployments, especially in complex and sensitive environments.

Appendix