Presenters
Source
From Data Scientist to Open Source Hero: My Grafana Plugin Journey 🚀
Hey everyone! Hamed here, and I’m thrilled to share my adventure as an open-source contributor to Grafana. It’s been a wild ride, transforming from a data scientist to building tools that empower others. Let’s dive in!
Who Am I and How Did I Get Here? 🤔
I kicked off my career about 10 years ago as a data scientist. Over time, I found myself increasingly drawn to the world of data engineering. My journey took me to Snapp!, the largest ride-hailing company in the Middle East, and now I’m contributing to innovation at O+X in Sweden. This path has given me a unique perspective on how data flows and how we can best visualize and interact with it.
Demystifying Grafana Plugins: Your Gateway to Customization 💡
Before we get into my personal story, let’s quickly understand what Grafana plugins are and how they extend Grafana’s already impressive capabilities. Plugins generally fall into three main categories:
1. Panel Plugins 🖼️
These are your go-to for visualizing data directly within Grafana. Think of those beautiful graphs, gauges, and charts you see – they’re all powered by panel plugins. To build these, you’ll need a solid understanding of TypeScript and React, as they live on the frontend.
2. Data Source Plugins 🔌
Want to connect Grafana to a specific database, message broker, or any other data store? That’s where data source plugins shine. These are typically backend plugins, requiring knowledge of Golang and the Grafana SDK. Of course, you’ll still need frontend skills in React and TypeScript to build the user interface for configuring these data sources. Connecting to Prometheus is a classic example of what these plugins enable.
3. App Plugins 🧩
These are the powerhouses that extend Grafana’s overall functionality. App plugins can bundle multiple panel and data source plugins, or even introduce entirely new features. The popular Kubernetes, Loki, and Redis plugins are fantastic examples of app plugins.
My First Plugin: Bridging the Visualization Gap 🌉
Back in 2021, Grafana released Node Graph with version 8. A problem arose: there wasn’t a straightforward way to visualize data as a graph using data sources within Grafana. The only existing option, AWS X-Ray, was a commercial product, leaving a gap in the open-source community.
This was my cue! I decided to develop a plugin to fill this void. My first creation was the Node Graph API plugin.
The Path to Contribution: A Step-by-Step Guide 🗺️
If you’re inspired to become a Grafana contributor, here’s a roadmap based on my experience:
- Identify a Problem/Need: Look for areas where Grafana could be enhanced with a plugin.
- Research and Validate: See if a solution already exists. If not, engage with the community on Slack or forums, and consult with Grafana developers. They are incredibly supportive and can help shape your idea.
- Choose Your Plugin Type: Decide whether you need a panel, data source, or app plugin.
- Develop: Leverage the excellent SDKs Grafana provides to bring your plugin to life.
- Validate: Use the Grafana plugin validator to ensure your plugin meets quality standards.
- Submit: Submit your plugin for review. Grafana’s reviewers will check its functionality and adherence to guidelines.
- Community Impact: Once approved, your plugin lands in the Grafana marketplace, ready to be used by thousands!
My Second Plugin: Unlocking Apache Kafka’s Potential 🐺
Next up, let’s talk about my second plugin: the Kafka data source plugin.
Understanding Apache Kafka 🌐
Before diving into the plugin, let’s quickly touch upon Apache Kafka itself. For those unfamiliar, Kafka is a distributed, fault-tolerant, and highly scalable pub-sub message broker system. Its key features include:
- Real-time data streaming: Handle data as it flows.
- High throughput: Process massive amounts of data efficiently.
- Fault tolerance: Designed to withstand failures.
- Scalability: Easily grow to meet demand.
Kafka is invaluable for event-driven architectures, data pipelines, and real-time analytics. Its architecture involves producers sending messages to a broker cluster, which then makes them available to consumers. From version 5 onwards, Kafka no longer requires ZooKeeper, and it’s now based on KRaft.
How Kafka Works: Producers send messages to topics on Kafka brokers. Topics are divided into partitions, a key feature for scalability. Messages with the same ID consistently land on the same partition. Consumers, organized into consumer groups, pull messages from topics. This allows for both scaling consumers within a group (up to the number of partitions) and broadcasting messages across different services via multiple consumer groups.
The Challenge: Seeing Beyond Kafka Metrics 📊
Around four years ago, while Grafana could display Kafka cluster metrics (like node count, topic numbers, message counts) through Prometheus and exporters, it couldn’t show the actual data residing within Kafka topics. This was a significant limitation for anyone wanting to visualize real-time Kafka data in Grafana dashboards.
The Solution: The Kafka Data Source Plugin 💡
My idea was to build a data source plugin that directly connects Grafana to Kafka, enabling visualization of the data itself. This plugin allows you to:
- Connect to your Kafka cluster: Simply provide bootstrap servers (whether on-premise, cloud-hosted, or SaaS).
- Visualize messages: Access and display your Kafka messages directly within Grafana.
Key Features of the Kafka Plugin ✨
This plugin is packed with features to make interacting with Kafka data a breeze:
- Real-time Monitoring: It’s completely streaming, eliminating the need for manual refreshes.
- Partition Querying: Dive into specific partitions.
- Autocomplete: Effortlessly search through numerous topic names.
- Flexible Offset Options: Choose from
latest,last N messages, orearliest. - Timestamp Options: Use Kafka’s event time or the dashboard’s receive time.
- Advanced JSON Support: Handles flat, nested, arrays, and mixed JSON types.
- ProtoBuf and Avro Support: Works with both schema registry or inline schemas.
- Authentication & Encryption: Supports SASL, SSL, TLS, mTLS, and SCRAM.
Case Studies: Putting the Plugin to Work 🛠️
Let’s look at some real-world scenarios where this plugin shines:
- IoT Sensor Monitoring 🏠: Imagine two buildings with legacy sensors, each using different schema versions for their data. Kafka ingests these metrics, and a schema registry stores the various Avro versions. Your Grafana plugin, using the Kafka data source, can decode these Avro messages and display live sensor data (temperature, humidity, etc.) in Grafana.
- Analytics Dashboards 📈: For app events like user actions, clicks, or purchase counts serialized in JSON and sent to a Kafka topic, the plugin’s JSON decoder can help you build powerful analytics dashboards in Grafana, whether or not a schema registry is involved.
Live Demo: Kafka in Action! 🎬
Seeing is believing! Let’s explore how the Kafka plugin works in a live demonstration:
Finding the Data Source: In Grafana, navigate to “Data Sources,” click “Add connection,” and you’ll find the “Kafka” plugin. It’s been downloaded over 4 million times!
Configuration: Once added, you configure your bootstrap servers, security protocols (plaintext, SSL, SASL), and authentication mechanisms (SCRAM, username/password, etc.).
Exploring Data: You can produce test messages in JSON format and see them appear instantly. The plugin supports various message formats (JSON, Avro, ProtoBuf, plain text) and offset options.
Live Dashboards: I’ve set up live dashboards showcasing Kafka data:
- Transaction Value: Visualizing transaction values, authorization outcomes (pending, authorized, failed). The green dot indicates a live, streaming feed – no refreshes needed!
- Transactional Ledger: Data in Avro format.
- Payment Analytics: Data in ProtoBuf format.
- Plain Text Data: Simple text messages.
You can leverage Grafana’s transformation capabilities to visualize the data exactly as you need it.
Schema Registry Integration: The plugin seamlessly integrates with schema registries for Avro and ProtoBuf. You can even test the connection or provide inline schemas if you’re not using a registry.
Future Works: What’s Next? 🚀
The journey doesn’t stop here! We’re actively working on exciting new features:
- Data Querying: This will allow you to query historical Kafka data, effectively enabling “time travel” to specific messages and offsets from the past.
- Alerting: Imagine defining alerts directly on Kafka topics! If a specific pattern emerges in a Kafka topic, Grafana will be able to alert you. This will significantly enhance proactive monitoring.
- Enhanced Authentication: We’re adding support for more authentication methods like Kerberos and OAuth to cater to broader community requirements.
Thank you for joining me on this journey! You can connect with me via the QR code, find the plugin repository on my GitHub, and even contribute by submitting issues or pull requests. Let’s build the future of data observability together!
(Audience applauds)