Presenters
Source
Securing Your Software Supply Chain: A Deep Dive into Container Security 🔐
Welcome to “The State of the Art,” a GoTo Unscripted mini-series exploring the cutting edge of tech, from generative AI to platform engineering, new languages, and crucially, security. In this episode, we tackle the complex world of container security with Adrian Mouat, a DevRel Engineer at Chainguard, a company dedicated to crafting secure container images. Adrian, author of “Using Docker” and former chief scientist at Container Solutions, joins Charles Humble, a seasoned programmer, architect, CTO, and tech journalist, for an illuminating discussion.
Let’s dive into how we can build a more secure future for our software!
The Container Conundrum: Early Insights & Evolving Challenges 🚀
Adrian was an early adopter of containers, spotting their immense potential. He quickly realized Docker’s value for managing Python environments, finding it far cleaner and easier to share than traditional virtual environments. The ability to package an application with its exact dependencies meant others could build and use it without worrying about version conflicts.
However, this convenience introduced a new set of challenges. A long-standing issue with containers is the slowness of base image updates. Docker Hub, a fantastic innovation, led to widespread use of base images (like Debian, Ubuntu, Alpine). The problem? Updating these images requires a two-step dance: the operating system packages need updating, then the base image maintainers must release a new version. This often leaves container images with out-of-date software, posing significant security risks due to potential vulnerabilities and preventing access to new features. Adrian highlights that most Linux distributions were designed for long-lived virtual machines or physical hardware, not the ephemeral, frequently replaced containers we use today.
Scanners: A Double-Edged Sword ⚔️
Many companies invest heavily in container scanning tools like Snyk or Docker Scans. These tools work by indexing all software within an image (via package managers, binaries, npm, pip) and comparing it against databases of known vulnerabilities (NVD, GitHub Security Advisories, distro-specific advisories).
While useful, these scanners often generate a massive list of vulnerabilities, making it implausible for organizations to address them all. Adrian points out that the signal-to-noise ratio is often low; many reported CVEs (Common Vulnerabilities and Exposures) are not actually exploitable in a given application. This creates a dilemma: if a vulnerability is in your own application code, you can fix it. But if it’s in a base image you pulled, and you already have the latest version, what can you do? This challenge often leaves developers overwhelmed, leading to a lot of work with limited impact on actual security posture.
The Distroless Revolution: Shrinking Your Attack Surface 🤏
One of the most effective ways to enhance container security is to reduce image size. The less software an image contains, the fewer potential vulnerabilities and the less there is to keep updated. This simple yet powerful idea is the core of Distroless.
The Google Distroless project, co-founded by Chainguard’s CEO Dan Lawrence and CTO Matt Moore, aimed to strip a Debian base image down to the absolute minimum required to run a typical Linux application. While some applications (like statically linked Go binaries) can run on a “scratch” image with nothing, most need essentials like TLS certificates for HTTPS calls, temporary directories, time zone data, and other small Linux libraries.
The result? Distroless images are often just a few megabytes in size, a huge leap from 50-200MB traditional images. This significantly reduces the attack surface, leading to fewer CVEs (if kept up-to-date), faster startup times, and less data transfer. Chainguard has taken this concept to the next level, creating Distroless base images for a wide array of software. When scanning a Chainguard image, you should ideally see zero CVEs. While a few might appear (under investigation or without a known fix), the critical advantage is that these minimal images accumulate far fewer new CVEs over time compared to larger images, even if both start at zero. Chainguard also provides “not affected” advisories, clarifying when a reported CVE (e.g., a Windows-specific vulnerability) doesn’t impact their Linux-based images.
Wolfi: Building Security from the Ground Up 🛠️
Chainguard’s commitment to security runs deep, literally. They developed Wolfi, their own Linux distribution, as the underlying operating system for their images. The core principle here is to build everything from source – right down to the Glibc level. Adrian argues this is the only true way to produce genuinely secure packages.
Building from source allows Chainguard to be incredibly granular, splitting packages to ensure users only pull in what they absolutely need, leading to smaller APKs (Alpine Package Keepers) compared to larger Debian packages. To assemble these Wolfi APKs into container images, Chainguard uses an open-source tool called apko, rather than traditional Docker build. This meticulous, source-based approach is fundamental to overcoming persistent security challenges.
Beyond Updates: The Power of Immutability ✨
When it comes to updating Chainguard images, the recommendation is clear: replace, don’t update. Just as Docker revolutionized configuration management by allowing you to wipe out and replace containers, ensuring consistency, Chainguard advocates for this immutable approach. This means you don’t run apk update within a running container; you deploy a brand new, updated image.
Adrian recounts how Google, for example, operates with a “build horizon.” Even if a microservice’s code hasn’t changed for weeks, they regularly pull down and replace the service with a newly built image. This ensures that any underlying libraries, utilities, or operating system components are constantly refreshed and up-to-date, preventing the accumulation of vulnerabilities in long-running instances. This shift towards immutability represents a new way of thinking for many in the industry, moving away from the “patch-in-place” mentality.
S-bombs & Attestations: Knowing What’s Inside 🔍
S-bombs (Software Bill of Materials) are essentially a comprehensive list of all software components, their versions, and transitive dependencies within an image. Standards like SPDX and CycloneDX define these often verbose XML documents. Chainguard has a distinct advantage in S-bomb creation: because they build everything from source, they can generate S-bombs at build time, knowing precisely what goes into each image. This contrasts with many other tools (like Grype’s Sift) that create S-bombs after the software is built.
Adrian draws an analogy to the food industry’s ingredient lists and traceability. The goal is similar: if a vulnerability is discovered in a specific software component (like Log4j), S-bombs could theoretically help identify all instances where that component is used. However, Adrian notes that the usefulness of S-bombs is currently limited due to a lack of widespread adoption, inconsistent tooling, and the challenge of getting S-bombs for all third-party software.
Attestations go beyond S-bombs. An attestation is a verifiable proof that something happened or is true. For containers, this includes image signatures (proving the image’s origin and integrity), build platform details, and confirmation of passed tests. Attestations are vital for establishing provenance, understanding the complete history and trustworthiness of a software artifact.
Defense in Depth & Essential Security Practices 🛡️
The overarching principle in security is defense in depth: never rely on a single layer of protection. Adrian illustrates this with a memorable anecdote about a university site hack where a hosting company claimed security due to a firewall, yet an ancient PHP version provided an easy backdoor. The “Swiss cheese model” analogy also applies: individual layers (slices of cheese) have vulnerabilities (holes), but by stacking multiple, diverse layers, you eliminate a single path through. S-bombs, in this context, contribute more to post-incident analysis than direct attack prevention.
Adrian offers several practical recommendations for bolstering container security:
- Immutability: Make file systems read-only to limit attacker actions.
- Sign Container Images: Tools like Sigstore allow easy signing, verifying images haven’t been tampered with.
- Avoid Running as Root: This is often the first thing attackers exploit.
- Capabilities & RBAC: Implement fine-grained access controls.
- Smaller Images: Reduce the attack surface and limit “living off the land” attacks where attackers use existing tools (like
sed,awk,chmod) within the image to escalate privileges. - Scan for Secrets: Tools like TruffleHog can identify accidentally checked-in secrets in images or Git repositories.
- Short-Lived Credentials: Avoid long-term credentials. Use OIDC where possible, or ensure credentials expire quickly (e.g., hourly instead of yearly).
Lessons from XZ Utils: The Supply Chain Challenge 🔗
The recent XZ Utils attack (often pronounced “X-Z”) is a stark reminder of the sophistication of supply chain attacks. This three-year campaign, orchestrated by a user named Jia Tan, involved gaining trust within the XZ Utils open-source project to insert a backdoor. A critical aspect of the exploit was that parts of the malicious code were only present in the uploaded binary or release process scripts, not always in the public source code repository.
Adrian explains how Chainguard’s approach helps mitigate such threats: by building everything from source (using Git checkouts rather than relying on tarballs), they would bypass the malicious components injected only into the release artifacts. Chainguard’s new Libraries product (e.g., for NPM) also builds everything from source, preventing attacks that rely on malicious packages published to registries but not present in the source repository. While XZ Utils was a C library (outside Chainguard’s current libraries offering), this approach highlights a robust defense against similar supply chain compromises.
Dive Deeper with Chainguard Academy 🎓
For those eager to explore further, Adrian recommends chainguard.dev and specifically edu.chainguard.dev (the Chainguard Academy site). Here, you’ll find a wealth of tutorials and resources on using Chainguard images, understanding Distroless images, and leveraging open-source security tools. It’s a fantastic resource for customers and anyone interested in enhancing their software security knowledge.
Thank you for joining us on this insightful journey into container security! Stay secure and keep building a better future!