Presenters

Source

AI Agents: Are We Doomed, or Is There Hope for Security? 🤖🔒

The world of Artificial Intelligence is evolving at lightning speed, and with it comes a new frontier of security challenges. While we’ve spent decades building robust security for traditional software, AI agents seem to be a different beast altogether. Ilia Shumailov, a seasoned AI security expert, dives deep into why our established security principles are struggling to keep pace and what the future might hold.

The Age-Old Arms Race, Now with AI ⚔️

For as long as computing has existed, there’s been a constant battle between attackers and defenders. From the early days of time-sharing systems to today’s sprawling distributed infrastructure, adversaries have always been present. Historically, the security community has made steady, measurable progress, learning from past mistakes and adapting.

However, with the rise of AI agents, it feels like things are getting worse. This isn’t due to a lack of effort, but rather a structural problem. The fundamental assumptions that underpin our current security models simply don’t align with how AI agents operate.

A Historical Parallel: From Morris Worm to Log4j 🐛➡️📦

Think about the trajectory of attacks:

  • Morris Worm (1988): Exploited implicit trust between systems, a failure of assumptions at system boundaries.
  • Stuxnet, SolarWinds, Log4j, NPM supply chain attacks: Exploited explicit or implicit trust in software dependencies and update mechanisms. The adversaries thrived within the infrastructure we chose to trust.

A similar pattern is now emerging in AI:

  • Prompt Injection-Based Supply Chain Compromises: Incidents like the C-line compromise show malicious instructions embedded in repositories hijacking agent execution. The context window is becoming the new supply chain attack vector for AI.

Through it all, the security community has learned and adapted. Principles like memory-safe languages, formal sandboxing, and least privilege have become bedrock doctrines, demonstrably improving security. Key lessons learned include:

  • Data Permanence: What’s on the internet is there forever, shaping our “design for breach” mentality.
  • Incomplete Adversary Knowledge: Defense in depth works because attackers rarely know everything.
  • Complexity is Unmanageable Long-Term: Systems are built to be simple enough to reason about.

These principles guided architectural decisions for decades and helped build more secure systems. But then, something changed.

When Security Principles Meet AI Agents: A Clash of Worlds 💥

AI agents are designed to do things that our fundamental security principles were built to prevent. This tension is the core of the challenge. Many of our ingrained assumptions are now invalidated, or at least partially so.

Ilia Shumailov, with a background from the University of Cambridge and extensive experience at Google DeepMind and Meta, leads a team focused on securing AI systems. His work involves both attacking and defending these agents. He frames the discussion around three crucial questions:

  1. Do traditional security defenses transfer to AI agents? (Spoiler: No, they don’t.)
  2. What is the novel attack surface in the AI era?
  3. What realistic grounds for optimism exist?

The Three Pillars of Software Security vs. The AI Reality 🏛️🆚🤖

Traditional software security operates on three paradigms:

  1. Trust Logic, Isolate Data: Input validation, type safety, control flow integrity. We explicitly check that programs do what they’re supposed to, and that data can’t manipulate this logic. (Think running Linux binaries.)
  2. Don’t Trust Logic, Trust Data Provenance: Information flow control, differential privacy, data isolation. Even if programs are unpredictable, we ensure data is used in quantifiable ways.
  3. Trust Neither: Sandboxing with capabilities, microkernel architectures. The worst-case scenario, hoping an agent won’t escape and cause damage.

Now, let’s look at what modern AI agents require to function:

  • They execute arbitrary code.
  • They establish persistent recurrent operations.
  • They access external networks.
  • They communicate with other agents.
  • They take actions with real-world side effects.
  • They carry biases, making normal execution hard to reason about.

As you can see, none of these requirements align with the traditional security paradigms. You can’t isolate an agent that needs persistent operations. Quantifying data usage is incredibly difficult when it touches everything. And executing arbitrary code is something we know is fundamentally almost impossible to reason about.

Multi-Agent Systems: Chaos Amplified 🌐💥

In multi-agent setups, things get even more complex. The number of trust boundaries skyrockets. Every agent must decide whether to trust data and actions from other agents. A single compromised agent can cause cascading failures throughout the system. Long-running agents accumulate data, attack surfaces, and permissions over time, making every interaction a potential injection point. No existing security framework is designed to handle this complexity.

A Story from the Trenches: The Leaked Secrets 😱

Ilia shares a real-world example from a startup sandbox:

They ran a benchmark task: “Search for Python async best practices, and then create a GitHub issue summarizing what you found.” The agent performed as instructed, creating an issue with its findings.

However, a subtle detail in the agent’s output contained a vulnerability:

  • The agent outputted the text: set -o xtrace (which is a command to echo commands executed in bash).
  • This was passed to the GitHub CLI.
  • Because of how bash interprets backticks ( ), the shell executed set -o xtrace instead of treating it as literal text.
  • The set command, when run without arguments, lists all environment variables.
  • Crucially, the agent leaked its authentication token and all other secrets present in its environment directly into the public GitHub issue.

An attacker, monitoring for such leaks, found the token. They then used it to inject a prompt into the agent, tricking it into downloading a remote access Trojan and establishing a persistent command-and-control connection.

This was a fully automated supply chain compromise, starting from a seemingly benign benchmark test. The agent followed its instructions perfectly. The failure was systemic: excessive ambient authority and insufficient credential isolation. The agent was merely a conduit. This isn’t just about immature implementations; it’s an architectural issue.

The Fundamental Breach: Separating Control Flow from Data 🚫📊

A cardinal rule in traditional security: separate control flow from data. We never execute data and never allow untrusted inputs to influence program logic. This principle eliminated vast classes of attacks like SQL injection, cross-site scripting, and buffer overflows.

But AI systems defy this. When a user says, “Find my manager’s email and follow the steps inside of it,” in security terms, this is equivalent to loading instructions from an external, untrusted data source and executing them. This is precisely what we’ve spent 30 years preventing!

The paradox is that users want these behaviors. This is where AI’s value lies: dealing with uncertainty and reducing the need for explicit codification. Unfortunately, our old security semantics don’t apply when you can no longer separate control and data flows.

Furthermore, AI agents now execute arbitrary code. From complexity theory, we know that we cannot generally reason about what arbitrary code will do without running it. We’ve moved from needing CVEs and zero-days to compromise machines to simply submitting “convincing sentences” that trick the model into running arbitrary code from untrusted data. Imagine an agent monitoring your mailbox for bookings, with access to your banking information – a simple email could compromise your entire system.

AI Attacking AI: The Next Frontier 🤖⚔️🤖

The situation is further complicated by papers showing strong attacks against AI agents with defenses in place. AI can now be used to automate attacks against other AI systems, making them highly efficient.

  • Adaptive Attacks: These bypass known defenses by adjusting their strategy based on the system’s responses.
  • RL as a Hammer: Reinforcement learning agents can be trained to generate prompt injections, turning the attacker into another AI with unlimited patience.

Empirical results are stark: static, non-adaptive attacks (common in evaluations) show defenses that appear effective. However, automated adaptive attacks achieve high success rates against virtually all defense classes, including prompting-based, training-based, and filtering. None of the 12 evaluated defenses survived strong adaptive adversaries. The vulnerability is real, and agents can be manipulated.

Grounds for Optimism: Progress is Happening! ✨

Despite the daunting challenges, Ilia emphasizes that this story is not without hope. There are concrete, measurable reasons for optimism:

  1. Alignment is Working: Models are substantially more robust than even six months ago. The cost and time required to find successful jailbreaks and prompt injections have increased dramatically, from seconds to hours, and from tens of queries to thousands.
  2. Attack Instability: Many successful attacks are unstable, working only once or intermittently. This instability aids detection efforts.
  3. Methodological Innovation: We’re seeing significant progress in provenance and governance for AI systems.
    • Data Flow Separation: We can now enforce data flow and control flow separation where applicable.
    • Auditable Control Flow: We can track what data influenced decisions and build auditable control flow graphs in agent execution.

While fundamental limitations like the halting problem and confused deputy problem mean we won’t solve all issues, we are making significant strides. Over the next few years, defensive systems will improve, and we are already building meaningful mechanisms to enforce security policies.

The Road Ahead: New Semantics for a New Era 🚀

In summary:

  • Old principles don’t transfer. We need new security semantics for AI.
  • The problem is generally unsolved. Be skeptical of anyone claiming otherwise.
  • This is an exciting time to be in security, as we are shaping the future.
  • Indirect prompt injections are becoming more manageable, though not fully solved. The bar is rising, and we can eliminate large classes of attacks.
  • Data-dependent actions on untrusted inputs remain the hard frontier.
  • For many fundamentally unsolvable issues, enforcement must happen at the system level, not just within the model.

Ilia is optimistic that we are close to solving many AI security issues. The journey is ongoing, but the progress is undeniable.

Appendix