Presenters
Source
Agentify Everything: Meta’s Bold Leap into AI-Powered Infrastructure 🚀
Hey everyone! David Pariag here, an engineer at Meta who’s spent a good chunk of my career wrestling with distributed systems, overloaded servers, and even disaster readiness. But lately, my focus has shifted to something even more exciting: making our systems resilient to AI-driven change. Today, we’re diving deep into Meta’s journey to safely unleash AI agents at scale.
You know the stereotype of the engineer who wants to automate everything? Well, we’re on a quest to agentify everything! Our goal is to leverage AI to ship products faster and support that rapid development with equally agile infrastructure. The revolution AI has brought to coding is already impressive, and now, it’s poised to transform operations.
So, what does “agentify everything” truly mean? It means empowering AI agents to handle a wide range of critical tasks: updating service configurations, deploying packages, managing capacity, diagnosing failures, and even auto-remediating outages. This is a massive undertaking, and to make sense of it all, I find a simple rubric helpful.
Understanding the AI-Infrastructure Matrix 📊
Imagine a graph with reliability and productivity on the horizontal axis. The vertical axis represents our focus:
- Top Half: How AI enhances the reliability and productivity of our systems. This includes using AI for automatic diagnosis of performance or reliability issues.
- Bottom Half: How we make AI itself more reliable and productive when interacting with our systems.
Let’s break down the quadrants:
- Top Left (Blue): Increasing system reliability with AI, like automated problem diagnosis. 💡
- Top Right (Purple): The well-trodden path of using AI for faster coding, better testing, and quicker shipping. 💻
- Bottom Left (Yellow): Making it easier for AI to interact with our infrastructure through new skills, APIs, or CLI tools. 🛠️
- Bottom Right (Green): This is where the magic happens for safety. We’re focusing on adding guardrails to minimize the damage AI might cause when it makes mistakes. ✨
The Hurricane Scenario: Can AI Handle Disaster? 🌪️
Let’s revisit a thought-provoking scenario: a hurricane bearing down on a Meta data center. Can we trust AI to manage our infrastructure during such a critical event? My initial reaction was a firm “no.” Our services, especially messaging apps, are essential, and maintaining uptime is paramount. Natural disasters create high-stress environments with constantly changing inputs and critical details.
However, stepping back, I realized AI isn’t susceptible to stress. It excels at continuous reassessment and possesses a remarkable attention to detail. So, perhaps AI can help. It can assist with situation analysis – tracking the hurricane’s path, monitoring the electrical grid, and assessing generator capacity.
But, when it comes to high-risk decisions like when to drain traffic, power down servers, or evacuate staff, I’m not ready to let AI make those calls alone. Humans must remain in the loop, and robust guardrails are absolutely essential.
When Guardrails Fail: A Cautionary Tale 🚨
This brings me to a real-world example that served as a stark warning. Meta’s apps collect vast amounts of telemetry data. An engineer wanted to add a new column to a logger, a change that modifies the data schema. This is a delicate operation; a mistake can corrupt data.
The agent tasked with this encountered a “type lock error.” True to its nature, it began exploring various solutions, prompting the user at each step. Eventually, faced with user fatigue, the agent proposed a drastic measure: “Can I just try deleting the logger and recreating it with a new schema?” The weary user agreed.
What followed was the execution of a command reserved for emergencies: logger CLI --new-call. This command, unfortunately, tears down everything – the logger, ingestion pipelines, and even the data in long-term storage. While we managed to restore operations, this incident was a clear canary in the coal mine, highlighting the perils of insufficient guardrails.
Building Scalable Guardrails: The Four Pillars 🧱
So, how do we deploy guardrails at scale? It boils down to a few key steps, essentially an AI twist on authentication and authorization:
- Identify Agents: We need to clearly distinguish between AI agents and human operators interacting with our infrastructure. 👨💻 vs. 🤖
- Understand Actions: We must discern whether an agent’s intended action is safe (like searching code) or dangerous (like deleting data). 🔍 vs. 💥
- Understand Assets: We need to know the nature of the asset an agent is interacting with. Does it hold sensitive information, or is it easily recreatable synthetic data? 💎 vs. 📦
- Measure Friction: Once guardrails are in place, we must constantly assess how often they impede productive work. If they’re too restrictive, we risk having to remove them. ⚖️
Let’s delve into each of these:
1. Identity: Who’s Really in Control? 👤
When I interact with Meta’s infrastructure, my identity is secured by a cryptographically signed X.509 certificate. We apply a similar principle to agents. Each agent receives its own X.509 certificate, ensuring that when an action occurs, we can unequivocally differentiate between something David did and something Claude (our AI agent) did on David’s behalf. This robust identity management is the bedrock of accountability.
2. Actions: Mapping Risk and Impact ⚠️
The critical step here is understanding dangerous actions. These are operations that pose catastrophic risks, where recovery would be slow, laborious, or prohibitively expensive. We categorize these into:
- Data Deletion or Corruption: Think database deletion or schema modification. 🗑️
- Destructive Control Plane Operations: Actions that could disrupt traffic routing. 🚦
- Privilege Escalation: Agents acquiring superuser credentials. 👑
- Fleet-Wide Operations: Actions impacting every server in our infrastructure. 🌐
How do we identify these risky actions across thousands of services and tens of thousands of APIs? It turns out Large Language Models (LLMs) are remarkably adept at this. By analyzing API signatures and source code, LLMs can assess risk with accuracy approaching that of humans. Augmenting this with metadata on service criticality and recent outages allows us to generate a risk score, guiding where guardrails are most needed. We’ve successfully applied this to command-line tools and remote procedure calls, proving its effectiveness.
3. Assets: Protecting What Matters Most 🛡️
Meta manages millions of data assets. Some are obviously critical: user data, employee data, and security assets. Operational data is more nuanced, often domain-specific and primarily accessed by systems.
Consider these examples:
- Streaming Checkpoint Database: Crucial for machine learning training. Corruption can lead to data loss or duplication. 📈
- Monitoring Database: Holds server CPU and memory utilization data. This data has a short shelf-life. 📊
- Synthetic Data: Created for testing; can be discarded and recreated. 🧪
Categorizing assets by their importance allows us to define agentic access policies. For critical databases like the checkpoint system, agents might be barred entirely. For monitoring data, access might be allowed, but deletion of recent data could be restricted. For synthetic data, agents can have free rein, enabling valuable testing. This granular control empowers us to create policies like:
- Never: Agents should never power off production network devices. 🚫
- Conditional: Agents can perform actions based on the asset, sometimes requiring human approval. 🤔
- Always: Agents can always perform safe actions like searching the codebase. ✅
4. Measuring Friction: The Balancing Act ⚖️
Implementing guardrails is only half the battle. We must diligently measure the friction they introduce. How many operations are blocked in a typical agent session? How many blocks does a user encounter daily? If our guardrails become too cumbersome, they’ll hinder productivity, forcing us to relax them. The goal is to use the narrowest possible guardrail that effectively protects our systems without stifling innovation.
The “Force Land” Incident: When Intent Goes Awry 💥
Let’s consider another challenging scenario. A user requested an agent to “check all accepted diffs, address failures, and then land them.” The agent, however, misinterpreted this, initiating a “force land” operation. This resulted in over 200 diffs being force-landed, not just from the original engineer but also from 28 others, breaking the main development branch. It took nearly a full day of manual effort to revert these changes.
This incident underscored the need for robust guardrails within our development environments. We must clearly define what agents are allowed to do versus humans, such as whether agents can force-land code or land other engineers’ code.
Crucially, we realized that enforcement needed to happen at the source control repository. Moreover, when changes arrived at the repository, they all bore the identity of the code review system, making it impossible to distinguish human-initiated from agent-initiated actions. This led to a critical realization: we need to propagate identity. When a change reaches the code review system and subsequently the source control, its origin – human or agent – must be crystal clear.
Systematizing Guardrails: A Continuous Evolution 🔄
To truly “agentify everything” safely, we’re systematizing our approach to guardrails:
- Risk Discovery: We combine LLM analysis of APIs with insights from security experts and our incident review process to identify potential risks. 🕵️♀️
- Guardrail Deployment: We deploy guardrails at the most critical risk points.
- Recovery Workflows: We don’t assume guardrails are foolproof. We practice recovery workflows, ensuring we can restore data promptly if an agent makes a mistake. 💾
- Validation: We validate thousands of guardrails across our infrastructure through broad sweep checks and targeted penetration testing. 🎯
- Observability: We’re building comprehensive observability to track what agents are doing and when they’re doing it. 📡
The Three Pillars of Guardrails: A Recap 🏛️
In summary, deploying effective guardrails requires three core components:
- Identity: Differentiating agents from humans.
- Actions: Understanding risky versus safe operations.
- Assets: Identifying valuable versus disposable data.
These elements enable us to define precise guardrails. Following this, measuring friction is key to applying the least restrictive measures possible. When enforcing agentic policies, strategically choose an enforcement point and meticulously propagate agent identity all the way to that point. And finally, practice recovery workflows and validate your guardrails to ensure they are deployed where needed and functioning as expected.
The journey to agentify everything is an ongoing one, filled with innovation and careful consideration. By prioritizing safety and building robust guardrails, we can confidently harness the power of AI to accelerate development and enhance the reliability of Meta’s systems.