Presenters

Source

Unleash the Power: Customizing Envoy Gateway with the Extension Server 🚀

Ever felt that a powerful open-source tool was almost perfect, but just needed a little something extra to fit your unique needs? If you’re nodding along, then this deep dive into Envoy Gateway’s extensibility is for you! We’re going to explore how you can go beyond the out-of-the-box features and truly make Envoy Gateway your own, all thanks to a game-changing component: the extension server.

Why Extensibility is King 👑

Let’s face it, in the fast-paced world of tech, extensibility isn’t just a nice-to-have; it’s a make-or-break factor for open-source projects. Companies like Confluent, who have been leveraging Envoy for years, initially had concerns about losing control over custom policies or resorting to maintaining forks. But their journey with Envoy Gateway revealed a crucial truth: its architecture is built for extensibility. This inherent design is what made their adoption not only possible but also incredibly powerful.

The Gateway API: A Solid Foundation 🏗️

Envoy Gateway stands tall as a robust implementation of the Kubernetes Gateway API. It masterfully translates Gateway API resources into Envoy’s powerful XDS configuration. But here’s where the magic truly happens: Envoy Gateway significantly extends the Gateway API by leveraging its built-in extension points. Think custom route filters for seamless rewrites, custom backend resources for those tricky Unix domain socket endpoints, and a whole suite of policies for enhanced security, traffic management, and data plane capabilities – like validating JWTs or implementing custom extension filters.

Enter the Extension Server: Your Gateway to Customization 🚪

At the heart of Envoy Gateway’s control plane extensibility lies the extension server. This pivotal feature empowers you, the end-user, to:

  • Craft your own extensions: Build and implement functionality tailored precisely to your needs.
  • Define custom extension resources: Create resources that seamlessly integrate with Gateway API’s extensibility points.
  • Supercharge the translation engine: Make essential XDS configuration changes to enhance how Envoy Gateway operates.

When to Call Upon the Extension Server? 🤔

The extension server shines in several key scenarios:

  • Tailoring Envoy Gateway for Specific Needs: When a feature you require falls outside the scope of the core Envoy Gateway project, perhaps due to limited community support, security considerations, or conflicts with Gateway API definitions.
  • Tactical Feature Adoption: Implement features rapidly via an extension server while simultaneously contributing them to the open-source project. This allows for faster access and eventual convergence with the main project.
  • Large-Scale Extension Projects: For ambitious undertakings that significantly extend Envoy Gateway, such as integrating AI-oriented APIs.
  • Vendor-Specific Distributions: Enable unique differentiating capabilities and custom features for your commercial offerings that aren’t present in the open-source Envoy Gateway.

Extending Envoy Gateway: Options & Recommendations 🛠️

While there are multiple ways to extend Envoy Gateway, the best approach depends on your needs:

  • Forking: Highly discouraged due to the significant maintenance overhead and potential for divergence.
  • Envoy Patch Policy: A good fit for simpler use cases that don’t involve complex logic or state management.
  • Extension Server: The recommended solution for substantial extension requirements, especially those involving state management, context-aware logic, and custom code implementation.

Core Concepts of the Extension Server 💡

Let’s demystify some of the key components of the extension server:

  • gRPC Service: The extension server communicates via a gRPC service. It implements an interface defined by Envoy Gateway and can connect via TCP or Unix sockets, with optional TLS/mTLS for secure communication. Its deployment is flexible – it can run as a sidecar, a separate deployment, or even out-of-cluster.
  • Interface Hooks: Envoy Gateway provides a set of “hooks” at various points to mutate XDS resources. These include routes, virtual hosts, listeners, clusters, and secrets. When Envoy Gateway encounters a registered hook, it invokes the extension server, providing translated XDS resources for potential modification.
  • Extension Server Resources (CRDs): Developers can define their own Kubernetes Custom Resource Definitions (CRDs). Envoy Gateway watches these CRDs, making them available as context when invoking extension server hooks. This enables context-aware mutations:
    • Policies: These can be attached to gateways with target references for specific scopes. They serve as context for HTTP listener and post-translation hooks, like implementing a custom buffering policy.
    • Route Filters: These allow for route-specific features, such as request hedging, by referencing custom extension filters.
    • Backend Resources: These enhance backend references, enabling custom translation of backend references into Envoy clusters. They provide contextual information to the cluster mutation hook, for example, changing clusters to original destination clusters for external load balancing.

Real-World Magic: Integrating with SPIFFE/SPIRE for mTLS 🤝

To truly grasp the power of the extension server, let’s look at a compelling demo scenario: integrating Envoy Gateway with SPIFFE and SPIRE Agent for robust service-to-service authentication. This isn’t a native feature, making the extension server essential.

The Goal: Configure Envoy to securely retrieve a TLS client certificate (SVID) from a SPIRE Agent (running locally and exposing a SDS API over a Unix domain socket) and present it to backends for mutual TLS (mTLS).

The Transformation: The demo gracefully transitions from a basic TLS setup to a full-fledged mTLS integration, powered by the extension server.

The Custom Policy: A custom “backend mTLS policy” CRD is introduced. When applied, this policy signals Envoy Gateway to activate the extension server’s mutations.

The Mutations in Action:

  1. SPIRE Agent Cluster Addition: A hard-coded cluster configuration for the SPIRE Agent is seamlessly added.
  2. Upstream TLS Transport Socket Mutation: For existing backend clusters, the upstream TLS transport socket is intelligently mutated. This ensures it fetches the SVID from the SPIRE Agent via SDS, using a Spiffe ID derived from the custom policy’s trust domain and workload identifier.

The Implementation Journey:

  1. CRD Creation: The custom policy CRD is implemented using Go and controller-gen.
  2. Envoy Gateway Configuration: Envoy Gateway is configured to watch for the policy and invoke the necessary extension server hooks.
  3. Extension Server Implementation: A Go-based extension server is built, specifically implementing the postTranslateModify hook. This hook parses the custom policy and applies the required XDS mutations.
  4. Deployment & Verification: The extension server is deployed, and its logs are checked, along with Envoy’s config dump, to ensure everything is functioning as expected.
  5. End-to-End Validation: A curl command is used to test the echo server, confirming successful TLS and verifying the presence of the peer certificate (SVID).

Your Questions Answered! 🗣️

The session also addressed some excellent audience questions:

  • Contributor Friendly? The extension server is designed to be accessible to new contributors, thanks to improved documentation and the use of common technologies like gRPC and cube builder. However, a solid grasp of Envoy and XDS is a prerequisite.
  • Pluggable Providers (Config/Infra)? While Envoy Gateway has evolved to support local file system configuration, making config and infra providers pluggable without forking is an area ripe for future development. The team is open to exploring common use cases and enhancing extensibility at this level. The fascinating idea of dynamic injection of provider implementations via Golang plugins was also raised!
  • SPIFFE vs. Service Mesh? The demo beautifully illustrates how Envoy Gateway’s extension server can integrate with external systems like SPIFFE/SPIRE for mTLS when secrets aren’t readily available in Kubernetes secrets. This addresses scenarios where traditional service mesh or Kubernetes secret management might not be the ideal fit.

This session truly showcases how Envoy Gateway’s extension server empowers you to go beyond its default capabilities, enabling sophisticated customizations and integrations to meet your specific operational demands. The future of API gateways is flexible, and Envoy Gateway, with its extension server, is leading the charge! ✨

Appendix