Presenters

Source

Alloy’s OpenTelemetry Engine: What’s New and What’s Next 🚀

Grafana Labs is leveling up its observability game with significant advancements in Alloy, their open-source observability data collector. This year marks a pivotal moment as Alloy officially becomes a fully compliant OpenTelemetry Collector distribution, bringing a wealth of new capabilities and a future-proof architecture.

The Evolution of Alloy: From Collector to Standard-Bearer 🕰️

Alloy’s journey began two years ago, announced as a collector supporting both Prometheus and OpenTelemetry Protocol (OTLP) formats. Last year, the focus sharpened on unifying high-performance Prometheus pipelines with OTel Collector workflows within Alloy. Now, the narrative shifts dramatically: Alloy is officially recognized as a fully compliant OTel Collector distribution, boasting native YAML support.

Why the Big Shift? 🤔

The reasons behind this strategic pivot are multifaceted:

  • Community Alignment: The OpenTelemetry community has tightened its definition of what constitutes an OTel Collector distribution. To remain community-first and uphold open standards, Grafana Labs is ensuring Alloy fits this definition precisely.
  • Vendor-Agnostic Workflows: Customers increasingly demand vendor-agnostic solutions, seeking configuration languages that are transferable across different providers. Alloy’s new direction directly addresses this need.
  • Future-Proofing: By aligning Alloy with the evolving OpenTelemetry Collector, Grafana Labs aims for Alloy to inherit changes and advancements by default, minimizing future development effort and ensuring continuous compatibility.

What This Means for You ✨

This evolution is crucial because Alloy is Grafana’s most widely distributed and deployed software. As OpenTelemetry becomes the industry standard for telemetry data generation and export, Alloy’s new engine grants you native access to the broader OpenTelemetry ecosystem.

Introducing the OpenTelemetry Engine in Alloy ⚙️

The core of this transformation lies in Alloy’s new OpenTelemetry Engine.

Defining the “Engine” 🛠️

  • Default Engine: This is the engine most familiar to existing Alloy users. It’s a “Swiss Army knife” that supports a wide array of components across Prometheus, OpenTelemetry, Loki, and more. It uses Alloy’s in-house configuration syntax (.alloy files).
  • OpenTelemetry Engine: This engine is essentially the upstream OpenTelemetry Collector runtime embedded entirely within Alloy. It understands OTel components and OTel YAML configuration syntax natively. This means you get a native OTel experience directly from Alloy.

Crucially, both engines live side-by-side within Alloy, offering unparalleled flexibility.

What’s Inside the OTel Engine? 📦

The OpenTelemetry engine, released with experimental stability in version 1.14, brings several key features:

  • otel Sub-command: Similar to Alloy’s run sub-command, this allows you to define, validate, and execute OTel YAML configurations. For those familiar with the upstream OTel Collector CLI, this experience will feel very familiar.
  • Alloy Engine Extension: This innovative feature allows you to run an instance of the default engine pipeline alongside the OTel engine pipeline from a single Alloy instance. You can run both in parallel!
  • Embeddable Health Dashboard: Essential for production deployments, this dashboard helps you monitor collector performance, resource usage, and data flow.
  • OpenTelemetry Collector Builder (OCB) Foundation: The OTel engine is built using OCB, an upstream tool that declaratively defines components and generates the collector logic. This ensures automatic upstream code integration and adherence to OTel component standards, making them accessible to any OTel Collector distribution.

Benefits and Migration Paths 🛣️

The introduction of the OpenTelemetry engine offers significant advantages:

  • Backwards Compatibility: If your current default engine setup works well, the OTel engine is opt-in. You can use the extension to try it out in parallel or in separate deployments.
  • Easy Migration:
    • Users of upstream OpenTelemetry Collectors with OTel YAML can easily point their configurations to Alloy’s OTel engine.
    • Conversely, users of Alloy’s OTel engine can migrate to other OTel Collector distributions if needed.
  • Custom Builds: Leveraging OCB and Alloy’s open-source nature, you can easily customize Alloy by editing manifest files. This allows you to remove, add, or even develop your own components to tailor Alloy to your specific use case.

A Real-World Demo: Engines in Harmony 🎶

A live demonstration showcased Alloy running a single instance with both the default and OTel engines enabled simultaneously.

  • The OTel engine pipeline ingested spans and traces from local services via an OTLP receiver, batching and sending them to a backend.
  • The default engine pipeline utilized Beyla, a feature that auto-instruments applications on the network. It scraped targets and wrote data to the backend, demonstrating the power of Prometheus-native pipelines and performance optimizations like Remote Write v2.

This highlights the core takeaway: you can pick and choose the strengths of either engine for your specific needs. Grafana Labs internally uses Prometheus-native pipelines for infrastructure monitoring due to their performance and OTLP for distributed tracing due to its rich metadata. Now, Alloy allows you to seamlessly combine these.

The demo also included a glimpse into Alloy’s performance at scale, processing over 12 billion spans in a 24-hour period on a large dev cluster, demonstrating robust scaling and stable export rates even with billions of data points and thousands of services.

Fleet Management: Orchestrating at Scale 🚢

Managing a large fleet of telemetry collectors can be a significant challenge. Grafana’s Fleet Management solution addresses this by providing a centralized control plane for Alloy collectors.

  • Key Capabilities:
    • Standardized Configuration: Create configurations once and deploy them to any number of collectors.
    • Controlled Rollouts: Intelligently distribute configurations based on collector attributes.
    • Dynamic Configuration Changes: Modify configurations on the fly, add/drop components, or enable/disable pipelines with a single click.

Fleet Management is already used by over 7,000 organizations, managing over 100,000 configuration pipelines and 360,000 active collectors.

Extending Fleet Management to All OTel Collectors 🌐

Grafana is extending Fleet Management’s capabilities beyond Alloy by implementing an OpAMP server. OpAMP (Open Agent Management Protocol) is the OpenTelemetry standard for remote agent management.

This means Grafana Fleet Management will soon be able to manage any OpenTelemetry Collector distribution, not just Alloy. This solution, currently in private preview, will be completely free to use.

The demo showcased creating an OTel configuration pipeline in Fleet Management to pick up logs from a local service and send them to Grafana Cloud. This was achieved remotely, without manual intervention on local configuration files or collector restarts, demonstrating the power of centralized management.

The Future is Open and Unified 🤝

Grafana Labs is deeply committed to embracing OpenTelemetry standards. For Alloy, this means solidifying its role as a compliant collector distribution and providing a native OTel experience. For Fleet Management, it means extending its reach to manage any OTel Collector.

The guiding principles remain consistent: embrace open standards and avoid vendor lock-in.

The OpenTelemetry engine in Alloy is available now for you to explore. Grafana Labs is actively working on enhancing remote management of the OTel engine at scale and contributing default engine strengths upstream. Stay tuned for exciting developments!

Appendix