Presenters
Source
Beyond the Big Screen: Reclaiming Observability with Peripheral Vision 👁️✨
We’ve all been there. The endless wall of metrics, the glowing dashboards promising ultimate insight. For years, the prevailing wisdom was simple: more screen equals more observability. If we could just see enough data, in enough places, we’d never be blindsided, never lose our flow state. But what if that’s not the whole story? What if those giant screens are more about appearances than actual engineering solutions?
This post dives into why traditional observability approaches might be missing the mark and introduces an exciting, human-centric way to keep your finger on the pulse of your systems.
The Illusion of the Big Screen 🖼️📉
The idea that massive wall displays are there to help engineers solve incidents is, frankly, suspect. The pandemic forced many of us to work effectively from home without these colossal screens, rendering them somewhat moot. For many, these displays serve a different purpose: to comfort management or impress visitors. No engineer wants to be interrupted by a manager pointing at a red alert and asking “Why is this red?” This leads to an unspoken agreement: keep the “pretty” dashboards on the wall, and hoard the truly useful, albeit potentially ugly, dashboards for ourselves.
The reality in operation centers is telling: nobody is actually looking at the wall displays. They’ve become more like “system status art” than functional tools. This isn’t just a social phenomenon; it’s rooted in how our eyes are fundamentally wired.
The Science of Seeing: Saccades and the Fovea 🧠👀
Our visual perception hasn’t evolved for staring at static screens. Our eyes make rapid, jerky movements called saccades – among the fastest movements our eyes can make, reaching up to 700 degrees per second. We use these to hunt for food or escape predators. Here’s the kicker: these critical saccades only process a tiny fraction of our visual field – just 2%.
This is because the fovea, a small spot on the back of our retina, is packed with specialized cone cells. Beyond this tiny area, our ability to perceive detail drops off dramatically. This fovea, responsible for our sharpest vision, makes up a mere 0.16% of our entire retina – about the size of your thumbnail or the moon in the night sky.
The implication? While our eyes dart around every few milliseconds, our brain dedicates a whopping 50% of its visual cortex to processing only those 2 degrees of our field of view. Think of your awareness as a frame buffer, updated with just two degrees of visual information per clock cycle.
The Alert Paradox: Flow State Interrupted 🚨⚡
We’re often expected to build and evolve systems, diving deep into our work. Then, an alert hits. This jolts us out of our productive flow state, replacing focus with a surge of adrenaline and cortisol. We’re thrown into a fight-or-flight response, often leading to a “freeze.” The solution? We open a browser, if we bookmarked the right dashboard. If not, we might give up and resort to SSHing into servers to tail logs – a far cry from efficient troubleshooting.
Hacking the Model: Peripheral Vision to the Rescue 💡🎯
This is where we can hack the model. While our foveal vision is sharp but narrow, our peripheral vision is wide but low-detail. It’s wired to detect motion, sudden changes, and contrast differences – the very things that grab our attention. This is why your phone on your desk is more likely to catch your eye than a distant wall display.
The goal is to leverage this powerful peripheral awareness to reduce your mean time to detection and minimize interruptions and stress.
The Stream Deck Revolution: Inspired by Twitch 🎮🚀
How can we achieve this? By borrowing a trick from Twitch streamers: the Stream Deck. This is a programmable macro pad that allows you to control various applications and actions. Its appeal lies in productivity hacking, and it’s become a mature platform over the years. Notice the size of those 72-pixel by 72-pixel screens on its buttons? They’re about the size of your thumbnail – perfectly aligned with our foveal perception sweet spot.
While I’m not affiliated with Elgato (the manufacturer), their developer program is excellent. You can find thousands of apps for the Stream Deck. My own journey began with a desire to integrate Grafana dashboards. When a dedicated app wasn’t available, I hacked together a solution to display stats and gauge controls directly on the Stream Deck. This places critical information within your peripheral vision, ensuring you notice it.
Technical Deep Dive: Grafana, APIs, and Encryption 🛠️🔒
My initial implementation uses Grafana’s v1 HTTP API (soon to be deprecated). While Elgato’s apps are DRM-protected, I’ve taken extra steps to scrub tokens and add additional encryption for security. You can explore these details on the GitHub repository.
The setup is straightforward:
- Create a read-only service account in Grafana.
- Configure this in the Elgato software.
- The software uses the API to pull existing dashboards and controls, focusing on gauge and stack controls.
- You then select the panels you want to display.
- You can even shorten control names for better visibility.
Within minutes, you can have a personalized, peripheral observability setup.
Alternatives and the “Professional” Touch 💻🦾
Of course, similar results can be achieved with a Raspberry Pi and a touchscreen in kiosk mode, or with free software on a phone. Makers could even build custom solutions with an Arduino. However, for a professional, desk-friendly solution that looks good, the Stream Deck offers a compelling option.
A key feature allows you to tap on a metric that’s behaving unexpectedly. This action opens the specific Grafana dashboard, providing immediate context. While having too many controls on a Stream Deck might be impractical, the principle aligns with Miller’s Law, suggesting we should aim for five to seven controls to avoid overwhelm. The focus is on monitoring signals that indicate something needs attention.
Beyond Alerts: Embracing Curiosity and Context 🕵️♀️💡
This approach shifts us from reacting to adrenaline-spiking alerts to picking up trends from the corner of our eye. It fosters curiosity and encourages proactive investigation. Imagine being on those late-night calls during an outage – the ability to quote real-time stats without breaking eye contact is invaluable.
Previous attempts, like the MacBook Touch Bar, weren’t perfect models for this. During major outages, we often experience perceptual narrowing, getting stuck in a single line of thought. Interestingly, police are trained to combat this in high-stress situations by forcing rapid eye and head movements.
You’re Already Trained: The Art of Peripheral Awareness 🥋💪
While you might not be a police officer or a fighter pilot with a heads-up display, you’ve likely been training for this your entire life. Think about it: when you’re in a tense situation, you’re focused, but you’re also tracking your surroundings, knowing when to break off and grab resources. If you can see the pattern, you’re already ready.
This is about taking observability out of the realm of overwhelming dashboards and into the intuitive, efficient flow of our natural perception.
Everything discussed is available on GitHub. Go forth and play!