Presenters

Source

Supercharging Kubernetes Networking: How Cilium Endpoint Slices Conquer Scale! 🚀

Ever felt the pinch of Kubernetes scaling limitations? You’re not alone! Microsoft’s engineering wizards, Tamil Mani and Shrea, recently pulled back the curtain on a revolutionary feature that’s transforming Kubernetes networking: Cilium Endpoint Slices. Forget those agonizingly slow pod startups and API server meltdowns. This is the deep dive you need to understand how Cilium Endpoint Slices are not just fixing problems, but paving the way for truly massive Kubernetes clusters.

The Pain Point: When Scale Becomes a Bottleneck 😫

Cilium, a powerhouse for Kubernetes networking, relies on a brilliant concept: Cilium Identities. These are stable, 16-bit numerical identifiers assigned to pods, unaffected by the fleeting nature of pod IPs. This identity-based policy enforcement is a security game-changer, offering a more robust approach than IP-based rules.

The magic happens through Cilium Endpoints, created for every single pod. Each endpoint holds crucial info: the pod’s IP and its precious Cilium Identity. The Cilium agent then builds an IP cache to swiftly map IPs to identities, enabling seamless network policy enforcement.

But here’s where the rubber meets the road – or rather, where the API server starts to sweat. At scale, this architecture unleashes a torrent of demands on the Kubernetes API server. Imagine a cluster with 5,000 nodes and a staggering 200,000 pods. This translates to an astronomical 1 billion Cilium watches! Every Cilium agent on every node is diligently watching all other Cilium endpoints across the entire cluster. The consequences?

  • API Server Performance Suffers: Expect increased latency in request-response cycles and alarming spikes in CPU and memory usage.
  • Out-of-Memory Errors: In the worst-case scenarios, the API server can simply crash, bringing your cluster to a halt.
  • Scalability Stalls: Your cluster’s ability to grow and accommodate more workloads becomes severely restricted.

The Hero Arrives: Cilium Endpoint Slices to the Rescue! ✨

Enter Cilium Endpoint Slices, the elegant solution designed to break through this scalability barrier. The core idea is brilliantly simple: batch Cilium Endpoints into a more manageable Kubernetes object – the Cilium Endpoint Slice.

Here’s the breakdown of this ingenious approach:

  1. Slimmed-Down Endpoints: The original Cilium Endpoints, while powerful, packed more information than Cilium strictly needed for its core operations. Cilium Endpoint Slices introduce a “core” representation, shedding non-essential fields like individual security labels and focusing purely on the vital Cilium Identity number.
  2. The Power of Batching: A configurable setting in the Cilium Endpoint Slice controller determines how many core Cilium Endpoints are bundled into a single slice. By default, this is set to a neat 100 core Cilium Endpoints per slice. This is a crucial optimization!
  3. Namespace-Aware Grouping: Endpoint Slices are intelligently batched by namespace. This ensures that representations for pods residing within the same namespace are grouped together, further streamlining management.
  4. Smart Updates: The system is designed to be efficient, intelligently batching updates into the Endpoint Slice with the most available slots. This minimizes the number of API server updates required.

The Transformation: A Smoother, Faster Experience 💨

So, how does this translate into real-world impact?

  • Cilium Operator Takes the Helm: A new Cilium Endpoint Slice controller, residing within the Cilium operator, becomes the orchestrator. It diligently watches for Cilium Endpoints created by the agents and then masterfully creates and manages the Cilium Endpoint Slices.
  • Agent’s Smarter Watch: Instead of individual agents drowning in individual endpoint watches, they now watch Cilium Endpoint Slices. This is a monumental shift! In a typical setup, this drastically slashes the number of watches from the total number of pods down to roughly the number of pods divided by 100.
  • API Server Relief: This strategic batching and consolidated watching significantly reduces the volume of data that needs to be written to and read from the API server. The strain is dramatically eased!

Quantifiable Wins: Real-World Impact at Microsoft AKS 📊

The adoption of Cilium Endpoint Slices has yielded some truly remarkable results, as evidenced by Microsoft’s deployment on Azure Kubernetes Service (AKS):

  • Three Times the Node Scale: The cluster’s capacity to handle a greater number of nodes saw a threefold increase.
  • 50% Faster API Responsiveness: The latency of the API server’s responses experienced a substantial reduction.
  • 60% Quicker Pod Startups: The time it takes for new pods to become fully ready has been significantly cut down.

These improvements are particularly impactful in scenarios where the control plane was previously struggling under the weight of Cilium Endpoint management.

Key Insights from the Experts 💡

The presenters shared some powerful statements that encapsulate the essence of this innovation:

  • “Cilium identity is the core entity of the Cilium.”
  • “Cilium enforced network policy based on the Cilium identity and not based on the FML part ips.”
  • “This will eventually cause pressure on API server if your scale is very high.”
  • “Cilium endpoint slice comes and saves us.”
  • “We’ve slimmed down the amount of information that needs to be written to or from the API server etc. We’ve slimmed that down. We batch the updates.”
  • “In the best case scenario… Instead of getting a 100 updates for each create or a label change on those pods, we’re going to get a single update at the API server and a single watch back to the Cilium agents.”

Understanding the Trade-offs: A Calculated Move ⚖️

While Cilium Endpoint Slices are a game-changer, it’s important to acknowledge the inherent trade-off: propagation delay.

  • CES Queuing Delay: Updates to Cilium Endpoint Slices are batched at the operator and then reconciled at the API server. This introduces a queuing delay. In scenarios with high churn (think 100 pods per second), this delay can reach up to 1 second.
  • Endpoint Propagation Delay: Similarly, the time between observing a new endpoint and its reconciliation with the API server can also hover around 1 second.

The presenters rightly argue that this minor delay is a small price to pay for the substantial gains in API server and control plane performance. It’s a critical step in preventing overload and unlocking superior cluster scalability.

The Tech Stack Powering the Solution 🛠️

This innovation is built upon a robust foundation of cutting-edge technologies:

  • Cilium: The core networking solution, bringing its powerful eBPF capabilities.
  • Kubernetes: The ubiquitous container orchestration platform.
  • eBPF: The underlying technology enabling Cilium’s highly efficient network programming.
  • AKS (Azure Kubernetes Service): The proving ground where these enhancements were rigorously tested and deployed.
  • Cilium Endpoint Slice Controller: The new star player responsible for managing Endpoint Slices.
  • Cilium Operator: The host for the Endpoint Slice controller.
  • Cilium Agent: The essential daemonset running on every node.
  • API Server: The critical central component of the Kubernetes control plane.

Seamless Migration and Rollback: Zero Downtime Guaranteed! ✅

The migration process is designed with minimal disruption in mind, ensuring your cluster stays online:

  1. Operator First: Begin by rolling out the Cilium operator with Cilium Endpoint Slice enabled. This allows the operator to start creating Endpoint Slices while your existing agents continue to watch the older Cilium Endpoints.
  2. Agent Second: Next, roll out the Cilium agent configured to watch Cilium Endpoint Slices and cease watching individual Cilium Endpoints.

This carefully orchestrated two-step approach guarantees that connectivity is maintained throughout the entire transition. Rollback follows the exact reverse procedure, offering peace of mind.

Fine-Tuning for Peak Performance ⚙️

For those seeking to squeeze every drop of performance, Cilium Endpoint Slices offer several tunable configurations:

  • Dynamic Rate Limiting: Adjust the Queries Per Second (QPS) for propagating Endpoint Slice updates, which can be configured based on your cluster’s size.
  • Slice Sizes: Experiment with modifying the default batch size (100 core Cilium Endpoints). Smaller slices might reduce propagation delays, while larger slices could potentially fit more updates into a single slice for heavily loaded control planes.
  • Priority Namespaces: Annotate specific namespaces to ensure their updates are processed by the Cilium Endpoint Slice controller before others, giving critical workloads priority.

The Road Ahead: Even More Exciting Developments! 🌟

The team isn’t resting on their laurels! They’re actively working on further enhancements:

  • Operator-Driven Identity Management: The goal is to move identity management entirely to the operator. This will help eliminate potential race conditions and bolster overall security.
  • Eliminating Cilium Endpoints: The ultimate aim is to completely remove the older Cilium Endpoint objects, relying solely on Cilium Endpoint Slices. This will further boost control plane performance by simplifying the architecture.

Q&A Highlights: Your Burning Questions Answered! ❓

The session also addressed key audience questions:

  • Open Source Availability: Great news! Cilium Endpoint Slices are a core part of Cilium open source and have been stable since version 1.1.7.
  • Rollback Procedure: The rollback mirrors the migration: first, roll back the Cilium agent to watch Cilium Endpoints, then the operator.
  • Downtime: The recommended two-step migration process is designed to avoid any downtime.
  • Slice Size Optimization: The optimal slice size is nuanced and depends on your pod distribution across deployments and namespaces. While smaller deployments might suggest smaller slices, the benefit can be diluted if pods are scattered across many slices anyway.
  • Chart Installation: When upgrading via charts, a two-step parameter change mirroring the operator-then-agent rollout is the safest bet to prevent disruption.

Cilium Endpoint Slices are a testament to the power of thoughtful engineering in tackling the complex challenges of modern cloud-native environments. If you’re running Kubernetes at scale, this is a feature you absolutely need to explore!

Appendix