Presenters
Source
AI: Your New On-Call Partner for Unbreakable Systems 🚀
Remember that gut-wrenching feeling when a critical system goes down? Gaurav Metra, a Production Engineer at Meta, sure does. He vividly recounts a harrowing Wednesday morning on December 11th, 2024, when a single configuration change triggered a widespread outage, impacting all of Meta’s apps for a grueling four hours. This wasn’t just a technical hiccup; it was a “sev zero” incident, requiring hundreds of engineers across dozens of teams, leading to sleepless nights and lost roadmap progress.
This experience, while extreme, highlights a fundamental challenge in managing massive, complex systems: experts can’t be everywhere at once. The sheer scale of operations means thousands of incidents occur annually, each demanding significant engineering time and expertise. This is where AI steps in, not to replace engineers, but to empower them, transforming how we approach platform reliability.
From Reactive to Proactive: The Reliability Flywheel ⚙️
Gaurav’s team is pioneering a revolutionary approach: building AI agents to act as “co-pilots” for engineers. The vision? A self-reinforcing reliability flywheel with four key stages:
- Pattern Identification 💡: Leveraging vast amounts of incident data, LLM-based agents semantically embed and cluster incidents by root cause similarity. These clusters are then decomposed into sub-patterns, enriched with LLM analysis, and scored by operational cost – the time spent debugging and following up on incidents. This allows for ranking prevention investments based on the highest operational costs and public fallout. Engineers now validate these clusters, name patterns, and decide on investments, while the AI handles the heavy lifting of data analysis.
- Autonomous Investigations 🕵️♂️: Encoded expert skills and workflows are stored in a context store, managed by an orchestration layer. LLMs reason about what to investigate and the data to fetch, with tools providing all the necessary production signals from over eight data systems. The output is structured, including ranked root cause hypotheses, correlated evidence, and mitigation recommendations, all linked to source data – no hallucinations.
- Autonomous Mitigations 🛠️: With guardrails firmly in place, agents can eventually perform mitigations.
- Self-Healing Infrastructure 🦾: The ultimate goal is infrastructure that can prevent issues before they arise, making systems reliability-aware and agent-aware.
Crucially, engineers remain at the center of this flywheel, orchestrating its progress. Each turn of the wheel makes the next faster, driven by engineers continuously teaching the AI.
Context Engineering: The Breakthrough 🔑
The team discovered that context engineering is far more scalable and robust than traditional prompt engineering. Instead of optimizing initial prompts, they focus on what the model can see in its context. Knowledge resides in tools, skills, and workflows, allowing any agent to leverage them. Adding a new capability simply means adding a new skill, not rewriting a prompt.
During an investigation, an LLM planner selects the best workflow, loads its constituent skills, and initiates parallel data gathering. This process, which previously took manual effort of 30-45 minutes (or more at 3 a.m.!), is now compressed into minutes. The system then enters hypothesis refinement loops, allowing engineers to guide the AI based on their observations.
Imagine the December 11th outage being handled by an agent: it would query time-series data, identify anomalies across regions, analyze chart patterns, cross-reference config changes within specific time windows, and examine BGP process crash patterns through log analytics. This could have transformed hours of engineering effort and days of follow-up into a 15-minute investigation and a targeted rollback.
Platformization and Trust: Building for Scale 🌐
Meta is heavily focused on platformization, ensuring the right guardrails are in place. Domain teams encode their expertise (failure patterns, services, workflows), while the platform handles guardrails, orchestration, tool integrations, and context stores. This approach is far more robust than disparate point solutions.
The results in production are compelling:
- 60%+ reduction in time from detection to mitigation across over 1,000 incidents.
- Over 40% of the time, the agent’s first root cause hypothesis is correct, and even when incorrect, it helps rule out other possibilities, reducing Mean Time to Detect (MTTD).
- An aggressive target of 95% reduction in MTTD, aiming for under 30 minutes per incident.
A Maturity Model for AI Agents 📈
The team is building towards a maturity model for their agents:
- Level 1: Automated Audits 📊: Reliability audit agents produce failure patterns, taxonomies, and knowledge graphs updated from incident data.
- Level 2: Investigation Co-pilots 🧑💻: These agents perform auto-RCA and suggest mitigation steps, leveraging live incidents and mined patterns.
- Level 3: Autonomous Mitigation 🤖: Agents can perform targeted, reversible rollbacks, with strict trust boundaries.
- Eventual Self-Healing Systems ✨: Agents perform reliability-aware development, pairing production signals with invariants to prevent issues proactively.
Each level is earned by demonstrating trust and reliability at the previous one. Transparency is non-negotiable, with every conclusion traceable to specific queries and results. If an agent cannot cite its source, it remains silent. The ultimate measure of success? On-call engineers sleeping better.
Key Takeaways for the Future of Reliability 💡
- Encoding, Not Reinventing: Agents are powerful because of the workflows and skills they use.
- Context is King: What models can see in their context is far more important than the model itself.
- Engineers Set the Ceiling: We are the architects of these systems.
- Trust is Incremental: Autonomy is earned through demonstrated reliability.
- Platformization Over Point Solutions: Robust guardrails and infrastructure are paramount.
- Relentless Measurement: End-to-end time from detection to mitigation is the key metric.
- Quality Through Evals: Every skill has an evaluation, driving continuous self-improvement.
This isn’t science fiction; it’s happening live in production at Meta. The goal is to make incidents like the December 11th outage a distant memory, transforming our role from reactive firefighters to proactive architects of unbreakable systems. It’s not just about AI; it’s about humans teaching AI, moving up an abstraction layer, and ensuring that when we’re on call, we can finally rest a little easier.