Presenters
Source
Closing the Generative AI Evaluation Gap: Building Better Benchmarks with Smarter Data 🚀
Hey everyone! Alex Ratner here, co-founder and CEO at Snorkel. We’re a frontier lab dedicated to building datasets and environments for evaluating, tuning, and training AI. Today, we’re diving deep into a crucial challenge in the world of generative AI: closing the evaluation gap. This is absolutely central to improving how we observe and understand AI agents.
From Agent Observability to Meaningful Benchmarks 🎯
Observing AI agents is a massive undertaking. It involves tracking their telemetry, monitoring the vast surface area of actions they take, the tools they use, and the outputs they produce. But simply seeing what an agent does isn’t enough. We need to measure it. This is where benchmarks come in. Benchmarks are formal evaluations where we define specific tasks, set desired outcomes, and develop ways to grade the agent’s performance. Today, we’re zooming in on this critical piece: building better data to create better benchmarks and truly understand if our agents are working, where they’re working, and if they’re performing reliably and safely.
The Measurement Lag: AI’s Capabilities Outpace Our Evaluation Skills ⏳
For the first time in AI history, our ability to measure AI has arguably been outpaced by our ability to develop it. Remember the days of training models to detect cats or dogs? Creating a few hundred labeled examples was straightforward, and measuring accuracy on a test set was simple.
Today, the game has changed dramatically. Model capabilities are advancing so rapidly that we’re struggling to create good datasets and benchmarks to evaluate them at the frontier of their abilities. This leads to saturated benchmarks where agents might score 80-90%+ on specific tasks, yet falter in real-world scenarios. Our evaluations are lagging behind the complex use cases we’re already deploying agents for. Without robust evaluation, all the monitoring and logging infrastructure becomes useless.
The Three Pillars of Advanced Agent Evaluation 🏗️
To build better benchmarks, we need to understand the evolving complexity of AI agents. We’ve distilled this into three key axes:
- Environment/Input Complexity 🌐: This ranges from simple coding challenges to an agent operating within an entire miniature company, complete with codebases, tools, and even simulated human interactions.
- Autonomy Horizon ⏳: This refers to the length and dynamism of the steps an agent needs to take. We’re moving from quick, single-step answers to complex, multi-week projects that can involve changing objectives and constraints.
- Output Complexity ✍️: As agents tackle more complex tasks in richer environments, their outputs become increasingly intricate and difficult to judge. Simple unit tests are no longer sufficient; we need sophisticated rubrics and verifiers.
Even in seemingly saturated areas like software engineering, there’s exponential whitespace for evaluation. Benchmarks like HumanEval are becoming less challenging, while newer ones like ProgramBench show agent accuracy rates as low as 3%. This illustrates a fundamental truth: the double exponential growth in task complexity and surface area is far outstripping gains in model sample efficiency.
The Heart of the Matter: Data for Better Benchmarks 📊
So, how do we close this evaluation gap? The key lies in building realistic data and environments that keep pace with the frontier of agent capabilities. This means:
- Richer Environments: Simulating real-world settings with tools, documents, databases, and personas.
- Longer Autonomy Horizons: Creating tasks requiring hundreds of steps, not just one or two.
- Nuanced Outputs: Developing complex rubrics and grading keys to assess agent performance.
When we talk about “data” in this context, we mean the test questions for our AI agents. In the early days of generative AI, this was a simple prompt and answer. Now, it’s a complex task, a domain-specific environment, and intricate rubrics that provide feedback throughout the agent’s execution.
The Art and Science of Building Benchmark Data 🛠️
Building high-quality data and environments for expert agents is a rigorous process:
- Scoping and Calibrating 📏: Defining the data specification is arguably the majority of the product spec for an agent. This involves ongoing calibration of what types of data, tasks, environments, and good outputs are required.
- Building Environments and Tasks 🏗️: For example, a legal agent needs access to legal research tools and a context of legal documents. A clinical agent needs patient vignettes and EHR data.
- Running Agents and Calibrating 🏃: Executing agents within these environments helps refine the tasks and environments themselves.
- Developing Rubrics and Verifiers ✅: Creating the mechanisms to evaluate agent correctness and success.
- Reviewing and Iterating 🔄: Every component is reviewed for quality and to identify edge cases that might require adjusting the data scope.
- Packaging and Governance 📦: Ensuring the data is versioned, well-governed, and usable as a standard evaluation tool.
Inspiring Examples: Pushing the Boundaries of Evaluation ✨
We’re proud to support innovative benchmarks that are driving progress:
- Terminal Bench 💻: An open-domain benchmark for agents with full terminal access. It’s been instrumental in guiding the progress of coding agents and is designed to cover diverse agentic operations and failure modes. A good benchmark adapts its data distribution to these failure modes, offering a mix of topics, domains, and error areas.
- Slop Code Bench 📝: This benchmark tackles coding tasks where the specification changes during the development process. It addresses the real-world challenge of evolving requirements and prevents the buildup of “slop code.”
- Continual Learning Bench 🧠: Launched out of Berkeley, this benchmark measures an agent’s ability to improve over sequences of tasks, highlighting the importance of learning from past experiences.
The Future of Evaluation: A Call for Open Benchmarking 📢
The evaluation gap in generative AI is a critical and exciting challenge. The complexity of agent capabilities across input, autonomy, and output is expanding rapidly. The key to closing this gap lies in developing realistic data and environments.
At Snorkel, we’re committed to supporting this endeavor through our Open Benchmarks Grants Program. We’ve committed $3 million and are supporting around 20 benchmarks, with more to come. We’re eager to back creative, well-designed benchmarks from open-source and academic teams.
By building better data and environments, we can create better benchmarks, leading to more reliable, safer, and more capable AI agents. Let’s bridge that evaluation gap together!