Presenters
Source
Harness Engineering: Building Software with Humans Steering and Agents Executing 🚀
The world of software development is undergoing a seismic shift, and at the forefront of this revolution is Ryan Lopopolo, a Member of Technical Staff at OpenAI. For the past nine months, Ryan has been living and breathing a new paradigm: building software exclusively with agents. He’s a self-proclaimed “token billionaire,” not in the financial sense, but in his belief that everyone should have access to the power of AI models to accomplish complex tasks.
Ryan’s core message is revolutionary: implementation is no longer the scarce resource in software engineering. Code is free. This isn’t just a catchy phrase; it’s a fundamental redefinition of our roles and responsibilities.
The Dawn of the Agent Era 🤖
The last six months have seen coding agents rapidly evolve, taking on increasingly complex tasks with remarkable reliability. This surge in capability means we’re no longer constrained by the sheer act of writing code. The bottleneck has shifted from implementation to direction and orchestration.
Key Takeaways:
- Code Abundance: We have an unprecedented abundance of code to solve problems. The challenge is no longer generating code, but deploying AI capacity effectively.
- Shifting Skillsets: The focus for human engineers is moving towards systems thinking, system design, and delegation.
- The “AGI Pill”: Ryan urges us to embrace the belief that models can produce, maintain, refactor, and even delete code. Our role is to unblock these agent teams and humans steering them.
- You Are a Staff Engineer: With access to vast AI capacity, each of us can operate like a staff engineer, managing numerous concurrent agent team members.
The New Scarcity: What Truly Matters Now ✨
In this new landscape, what are the truly scarce resources? Ryan identifies three:
- Human Time: Our most precious commodity. The goal is to automate mundane tasks and elevate human involvement to higher-leverage activities.
- Human and Model Attention: Directing focus effectively is crucial.
- Model Context Window: Maximizing the utility of the information provided to the models.
The Impact of Free Code:
When code is free, even low-priority tasks (P3s) can be tackled immediately, potentially in parallel. This unlocks new possibilities, as demonstrated by Ryan’s experience building internal tools with excellent localization for international teams without sacrificing other development efforts.
The Art of Prompting and Guardrails ✍️
With implementation becoming less critical, the prompt and the guardrails become paramount. This is where human expertise truly shines.
Key Principles:
- Legible Systems: Build systems, software, and structures that are easily understood by agents.
- Native Structure: Structure your codebase in a way that’s native to agents.
- Respecting Context: Be mindful of the model’s context window and make information predictable.
- Consistency is Key: Standardize practices to reduce the cognitive load on the models. Large-scale refactoring is now effectively “free,” making consistency achievable.
Defining “Good” for Agents: Specifying Non-Functional Requirements 🎯
The complexity of writing high-quality, maintainable code often lies in the subtle, non-functional requirements that engineers internalize over years. Agents, having processed trillions of lines of code, have seen every possible choice. Our job is to specify these requirements clearly.
How to Guide Agents:
- Document Expectations: Clearly write down what constitutes an acceptable job.
- Refine and Restrict: If agents falter, refine their output by setting guardrails. This might involve short-term velocity hits to implement robust checks.
- Learn from Failures: Analyze recurring failures by both humans and agents to systematically eliminate classes of misbehavior.
Tools and Techniques for Agent Empowerment 🛠️
How do we ensure agents consistently deliver high-quality results? By actively “prompt injecting” them with the necessary context and guidance.
Practical Approaches:
- Agent Skills and
agents.mmd: Well-defined agent configurations are essential. - Auto Code Compaction: Advanced models excel at managing context, reducing the need for manual code input.
- Reviewer Agents: Implement agents that scrutinize code for specific requirements like security and reliability, acting as automated quality checks.
- Bespoke Lints: Create custom linting rules tailored to your codebase to enforce specific non-functional requirements (e.g., ensuring timeouts and retries on network calls).
- Testing Source Code: Write tests that assert the structure of the code itself, not just its syntax or behavior.
- Context-Efficient Engineering: Adapt your codebase to be context-efficient, squeezing more value from model capabilities.
- Informative Error Messages: Provide remediation steps within error messages for both humans and agents.
- Prompt Engineering: Leverage various methods for injecting prompts, including lint error messages, review agents, and even using agents to help write prompts.
The Future of Software Development: Orchestration and Automation 🌐
Ryan envisions a future where a given token budget can drive product advancement over extended periods, with minimal direct human intervention.
Key Aspects of the Future:
- Agent Self-Improvement: Agents can build their own tools to address gaps in their capabilities (e.g., QA smoke testing).
- Beyond Code Writing: The focus shifts to higher-level activities like triaging user feedback, managing incidents, and ensuring product quality.
- Meta-Programming: Writing down processes and acceptance criteria becomes the new “programming” for agents.
- Disposable Code: Code is viewed as a build artifact, akin to how LLMs are fuzzy compilers, with harnesses acting as constraint layers.
Getting Started with Coding Agents: A Practical Guide 💡
For engineers accustomed to manual coding, transitioning to agents requires a strategic approach:
- Build Confidence: Use agents to write more tests for your existing code. This improves code quality and increases agent reliability.
- Analyze Your Time: Identify where you spend most of your time (writing code, waiting for tests/reviews, CI).
- Automate Incrementally: Use agents to automate the parts where you spend the most time.
- Focus on Delegation: Shift your role to defining work, prioritizing, and orchestrating agents.
- Embrace Primitives: Build reusable “building blocks” that agents can leverage, similar to how you’d empower your human team.
Collaboration in the Agent Era 🤝
Collaboration primarily happens within the repository and platforms like GitHub, with PRs serving as a central hub. The key is to optimize for throughput and avoid blocking progress on minor feedback. Agents should have the agency to acknowledge, defer, or reject feedback, fostering a bias towards code acceptance rather than perfection.
Maximizing Your Token Budget 💰
For those venturing into paid plans, consider this advice:
- Balanced Usage: Allocate tokens across planning, ticket curation, documentation, and CI.
- CI is Crucial: Invest in spending tokens in CI, as code acceptance and product advancement are now key value drivers.
- Plan Mode with Caution: If using plan mode, rigorously review and approve plans as PRs to avoid encoding bad instructions.
The future of software engineering is here, and it’s about harnessing the power of intelligent agents. By shifting our focus from implementation to direction, and by meticulously crafting our prompts and guardrails, we can unlock unprecedented levels of productivity and innovation. The era of “harness engineering” has arrived, and it’s an exciting time to be a builder.