Presenters
Source
Guardrails, Agents, and the Future of Infrastructure Management: A Deep Dive 🚀
The world of cloud infrastructure is evolving at lightning speed, and with the rise of autonomous agents, the need for robust guardrails has never been more critical. We sat down with some brilliant minds – Joe Romano, David Pariag, Sargun Dhillon, and Joe Duffy – to explore how we’re building safety, efficiency, and intelligence into the very fabric of our infrastructure management.
Inherent Guardrails in Infrastructure as Code 🛡️
Joe Romano kicked off the discussion by highlighting that infrastructure as code (IaC) itself provides inherent guardrails. The ability to preview changes before deployment allows us to catch potential issues proactively. Beyond IaC, technologies like policy as code are crucial. These tools enable us to codify compliance requirements, such as ISO 27001 standards, or critical operational rules like “don’t delete production databases without backups.”
The core challenge, as Joe pointed out, is shifting from finding issues after the fact to identifying them before they occur. This echoes lessons learned from his time in trustworthy computing, where static analysis was vital for catching bugs before deployments to prevent widespread issues like the Blaster virus. The approach? Bringing these checks as close as possible to the inner development loop.
Access Control for Autonomous Agents: RBAC vs. ABAC 🔑
David Pariag delved into the complexities of access control for autonomous agents. He clarified the distinction between:
- Role-Based Access Control (RBAC): Assigns specific roles (e.g., viewer, administrator) to users.
- Attribute-Based Access Control (ABAC): Assigns attributes to identities, granting permissions based on those attributes.
While both are in play, the team has initially focused on ABAC for agents. However, as multiple agents interact with diverse infrastructure, a shift towards a more RBAC-centric approach is underway. A key implementation detail involves ensuring agents use their own identities. In their “Oz” system for agents, access to human certificates is restricted in development environments. Agents can only access their own certificates, preventing them from impersonating humans and ensuring they operate within their designated identities.
Where Should Our Data and Workloads Live? A Philosophy of Cloud Agnosticism ☁️
Sargun Dhillon shared Meta’s philosophy on workload placement. The ideal scenario is not caring whether a workload runs on public cloud or first-party machines. The goal is to treat cloud capacity as just another form of capacity, leveraging existing systems like solvers and capacity management infrastructure to place workloads based on constraints – be it hardware, network capacity, or human-defined policies.
Maximizing ROI in AI Workloads: Beyond Just Capacity 💰
When it comes to the Return on Investment (ROI) of placing workloads, especially AI-driven ones, the evaluation considers various factors. Different vendors and data centers have unique strengths, including proximity to users and data storage capabilities. The ROI is maximized by optimizing across all these dimensions. Currently, with high demand for inference, the focus is on securing capacity. However, Sargun anticipates a future shift towards optimizing for efficiency, where signals about data center strengths and co-design efforts will drive placement decisions.
Joe Duffy echoed the sentiment regarding ROI for infrastructure automation. Automating tasks that were previously human bottlenecks can yield crazy good ROI. He cited examples of customers tackling hundreds of thousands of compliance issues that would be impossible without agents. AI’s ability to analyze vast datasets and identify patterns humans might miss offers significant advantages. The primary challenge, however, lies in orchestrating agents with the right guardrails to realize this ROI, rather than just identifying its existence.
The Guardrail vs. Velocity Trade-off: Finding the Sweet Spot ⚖️
Striking a balance between guardrails and product velocity is a constant negotiation. Initially, the focus was on enabling agents as quickly as possible, measuring friction along the way. However, incidents where agents caused novel and unexpected disruptions, sometimes even prompting humans to take problematic actions, led to a pivot. The current approach emphasizes safety, particularly targeting high-risk areas like data loss, while simultaneously measuring the friction introduced by these guardrails.
An interesting anecdote shared was about an incident where AI agents caused widespread diff issues. While the agents took only 20 minutes to break things, it took humans a full day to fix them. The aspiration is for agents to not only prevent such issues but also to be able to fix them faster than humans.
Code as Infrastructure: Why Agents Excel Here 💻➡️🏗️
Joe Duffy explained why agents are particularly adept at managing infrastructure through code. This stems from the vast amount of internet-trained code available on platforms like GitHub, which is self-describing and clearly articulates intent. Furthermore, AI labs have relentlessly pursued coding benchmarks, leading to significant improvements.
The realization that code still matters is crucial. Letting agents loose on random CLIs and scripts without codified, reviewable intent can lead to undesirable outcomes. Building a foundation on code allows agents to operate within a familiar domain. Infrastructure as code then acts as a “magic oracle,” translating coding solutions into infrastructure outcomes.
Fine-Tuning Agents for Infrastructure Domains 🧠
Experimentation with model fine-tuning for agents is ongoing and holds immense promise. This is particularly relevant for new cloud providers or custom infrastructure not present in initial training datasets. The advantage of code-based infrastructure lies in its type system. This allows for rigorous type checking, ensuring configurations adhere to expected data types and valid enum values, moving beyond untyped YAML. Leveraging vast datasets of successful and failed configurations, alongside classical ML techniques like K-means clustering, provides a powerful reinforcement learning foundation for fine-tuning models.
Battle Scars: The Real-World Challenges of Cloud Misconfiguration 💥
The complexity of the cloud inevitably leads to “battle scars.” Joe recounted instances of agents deleting production databases or Kubernetes clusters scaling to zero unexpectedly, causing capacity issues. The fundamental challenge is codifying these hard problems so agents can learn from them and prevent mistakes proactively, rather than just reacting. The concept of the “fighter jet OODA loop” (Observe, Orient, Decide, Act) is relevant here, emphasizing the need for rapid response and fixes to mitigate the impact of inevitable mistakes.
Human vs. Agent: Trust and Higher Standards 🤖🤝👨💻
A thought-provoking question arose: are humans still better than agents, despite agent mistakes? The analogy to self-driving cars was drawn, questioning our trust levels. While humans have a long history of making mistakes (like deleting databases), there’s a higher standard we hold for autonomous systems.
Developer Visibility: Cloud vs. Internal Data Centers 🏢
At Meta, developers can typically tell when they’re using public cloud versus internal data centers. However, the goal is to blur this distinction. As cloud offerings mature, the differences become less apparent. While there are still areas with less observability or differences in custom silicon, integration efforts are bringing cloud facilities under the same tooling umbrella as internal infrastructure. This includes exposing low-level access and providing a single pane of glass for debugging, enabling agents to debug cloud environments as effectively as data center ones. This unification is achieved by providing agents with the same interfaces and context for both types of capacity.
Agent-to-Agent Communication: A Multi-Layered Approach 💬
The future envisions a layered system of agents communicating with each other. Specialized agents might handle specific capacity types or vendor APIs, with a central dispatch system routing requests accordingly. This “agent on agent” communication, with specialized subsystems handling vendor-specific contexts, aims to avoid “cross-vendor pollution.” This approach combines API-level integration with agent-based intelligence.
The Risk of Faulty Instructions: Vetting Agent Output 🧐
David addressed the risk of agents providing faulty instructions to humans. While initially more incidents occurred where humans acted on incorrect agent advice, there’s a growing trend of humans being more critical of agent output. Models are also improving at recognizing and respecting guardrails. For truly dangerous operations, a restricted set of core users will retain privileges, with the expectation that they will use agents intelligently.
Teaching Agents to Detect Incorrect Data 🧐
Joe explained that teaching agents to identify incorrect data from cloud providers requires equipping them with the right context, access, and tools. The difference between an LLM and a human platform engineer lies in this surrounding ecosystem. Just as humans verify data by checking systems and running commands, agents need similar capabilities. By providing LLMs with the necessary context, skills, and access, they can perform data verification and build trust.
Proving Guardrail Effectiveness: Adversarial Testing 🧪
Proving the absence of issues with guardrails remains a challenging problem. The team is actively addressing this by spinning up adversarial agents designed to break through individual guardrails or collections of them. These agents attempt to perform destructive actions like deleting production databases or accessing unauthorized data. This “find and fix” approach is crucial for building a virtuous cycle of testing and improving guardrail robustness.