Presenters
Source
Diving Deep into the IPv6-Only Kubernetes Frontier: ESNet’s Pioneering Journey 🚀
The world of technology is constantly evolving, and networking is no exception. As we push the boundaries of what’s possible with data and connectivity, the transition to IPv6 becomes not just an option, but a necessity. ESNet, a leading force in high-performance networking for U.S. Department of Energy research, has boldly stepped onto the IPv6-only Kubernetes frontier, sharing invaluable lessons learned from their ambitious journey.
The “Why”: Mandates and Modernization 🎯
ESNet, the backbone for national labs and user facilities, is all about breaking down geographical barriers to accelerate scientific discovery. Their commitment to open-source innovation, evident in contributions like BPF and Zeke, naturally led them to tackle the complex challenge of modernizing their vast data network with containerized applications.
Two major drivers propelled ESNet towards this cutting-edge adoption:
- Modernizing Operations: The need to manage their extensive data network with contemporary, containerized applications.
- The OM 21107 Mandate: A critical government directive compelling federal agencies to phase out IPv4. This mandate meant ESNet had to “eat their own dog food,” diving headfirst into the uncharted waters of IPv6-only Kubernetes.
Initial Hurdles on the Path to IPv6-Only Kubernetes 🚧
ESNet’s journey began with a dual-stack approach, gradually transitioning to an IPv6-only setup on-premises. This wasn’t without its bumps in the road, especially when bootstrapping K3S, their chosen solution for its robust, out-of-the-box capabilities in their small-scale use case.
Bootstrapping K3S: The First Big Challenge 💡
The K3S installer script, downloaded from GitHub, initially failed. The culprit? Not DNS, but DNS64. This crucial mechanism synthesizes IPv6 addresses for IPv4-only destinations, highlighting an immediate need for IPv6 translation services.
Underlay Infrastructure: The Foundation for Success 🛠️
ESNet quickly realized the critical importance of:
- Node-to-Node Connectivity: Ensuring seamless communication between Kubernetes nodes.
- External Registry Access: Reliable connectivity to pull container images from registries.
This underscored the necessity of DNS64 and NAT64 for bridging the IPv4/IPv6 gap and enabling communication with essential IPv4 resources like GitHub.
The NAT64 Gateway Solution: A Creative Workaround 🌉
To bridge this connectivity gap, ESNet ingeniously leveraged Cilium as a standalone NAT64 gateway. While not officially documented for this specific use case, running Cilium as a Docker container proved remarkably effective, with significant help from Daniel Borkman at Isovalent. A fascinating tidbit emerged: Cilium was originally designed with an IPv6-first approach, with IPv4 support added later, making it a natural fit for this challenge.
Kubernetes Networking: Beyond the Defaults 🌐
A paramount lesson learned by ESNet is the critical advice: “don’t simply rely on defaults.” The documentation for deploying IPv6 on Kubernetes is often sparse, demanding a DIY spirit and deep dives into GitHub issues and source code comments, especially for on-premises deployments where network design and traffic flow are the user’s responsibility.
Architectural Decisions: The Long-Term Impact 🧠
Key architectural choices have far-reaching consequences:
- IPAM Mode and Routing Mode: These are often one-time, impactful decisions that shape the network’s future.
- Non-Overlapping IP Addresses: Planning for distinct IP address spaces for pods and services is vital, particularly when considering future cluster mesh deployments that require unique IPs across multiple clusters.
Ingress vs. Egress: The Evolving Challenge 📈
While ingress (traffic coming into the cluster) was relatively straightforward with predictable IPv6 addresses for applications, egress (traffic leaving the cluster) in a multi-tenant environment presented significant complexities. The inherent unpredictability of pod scheduling and their IPv6 addresses made it challenging for firewalls to correlate traffic to specific applications. This was a notable hurdle with Cilium 1.15.
ESNet’s Evolving Architecture: Adapting for Predictability 🔄
To tackle these egress challenges, ESNet experimented with a dual-interface node design:
- ETH0: Handled north-south traffic.
- ETH1: Managed east-west traffic.
Nodes joined the cluster via ETH1 but served applications over ETH0. This offered a degree of predictability but still involved masquerading, which hindered traditional security teams’ ability to correlate logs. To overcome this, they pinned workloads to specific nodes using node selectors, ensuring predictable egress IPv6 addresses.
The “Holy Grail”: Cilium 1.18+ and Native Routing ✨
The release of Cilium 1.18 and subsequent versions marked a monumental leap forward, introducing what ESNet considers the “holy grail” of Kubernetes networking. The advent of multiple IPAM (IP Address Management) and native routing capabilities, which officially reached General Availability (GA) in Cilium 1.19, unlocks powerful new possibilities:
- True Microsegmentation: Each application namespace is now allocated its own /64 subnet. This facilitates fine-grained policy enforcement and robust workload isolation at both the cluster and network layers.
- Clean Ingress and Egress: The elimination of masquerading restores end-to-end visibility, drastically simplifying troubleshooting and security analysis.
- Simplified Development: Developers can now run IPv6-only clusters without needing to be IPv6 experts, making it significantly easier to test application compatibility.
Key Takeaways for Your IPv6 Journey 🔑
ESNet’s pioneering experience offers invaluable insights for anyone embarking on an IPv6-only Kubernetes journey:
- Underlying Network Readiness is Crucial: Ensure your network is prepared with DNS64, NAT64, and solid IPv6 address planning. For advanced features like multi-IPAM and native routing, BGP becomes essential.
- Cluster Network Trade-offs Exist: Understand the implications of pod/service side configurations, native routing, and overlay networks.
- Beware of Hardcoded IPv4: Developers often embed IPv4 addresses in Dockerfiles, configuration files, Helm charts, and application source code. ESNet combats this technical debt by providing IPv6-only remote development environments and enabling local testing with IPv6-only Kind clusters on macOS.
- Observability is Paramount: Hubble is an indispensable tool for troubleshooting within Cilium and Kubernetes. Be prepared for the sheer volume of flow and log data it generates, which can potentially overwhelm SIEM systems.
/64Subnets for Routers: While it might seem like overkill, using/64subnets per application is crucial for router efficiency, effectively utilizing TCAM space for summarization and faster lookups.
ESNet’s ongoing blog series promises to delve deeper into the intricacies of multi-IPAM and native routing. Their journey is a testament to the complexity and immense rewards of embracing IPv6-only environments, paving the way for a future of scientific progress unburdened by geographical constraints. 🦾✨