Presenters

Source

Beyond Prompt Engineering: Mastering Context Engineering for AI at Scale 🚀

In the rapidly evolving landscape of AI-driven development, the buzzword prompt engineering is starting to feel a bit like yesterday’s news. As we move from simple chatbot interactions to complex, agentic workflows, the real secret sauce is context engineering.

I sat down with Adi Polak, Director at Confluent and author of Scaling Machine Learning with Spark, to discuss how we can move beyond basic prompting to build sophisticated, stateful AI systems that actually get work done.

🧠 Prompting vs. Context Engineering: What’s the Difference?

While prompt engineering focuses on how we instruct a model (e.g., role assignment, few-shot examples, chain-of-thought), it often treats the LLM as a stateless tool. Context engineering shifts the focus to how we manage the information surrounding that model.

  • Role Assignment: While popular, this is becoming less critical as environments become more specialized.
  • Few-Shot Examples: By providing high-quality examples of good and bad outputs, we teach the model patterns without needing to retrain it.
  • Chain-of-Thought: Modern models now perform this autonomously, breaking down tasks into logical steps—a fascinating leap toward self-correcting AI.
  • Constraint Settings: Providing strict technical specifications ensures the model builds software exactly as required.

🛠️ The Shift to Agentic Workflows

The true game-changer is moving from a human-centric “ask and receive” model to an agentic workflow.

Adi shared a powerful example: she recently spent 4 hours struggling with a complex Git operation due to context switching and mental fatigue. By using an agentic tool (Claude Code), she accomplished the same “surgical” task in just 5 minutes.

The lesson? We aren’t just using LLMs to write text; we are using them as agents that can execute workflows. The key is state management. Instead of starting from scratch every time, we should save successful workflows as skills.

🏗️ Architecting for Context: The “Skills Library” Approach

To scale effectively, engineering teams should build a repository of skills.

  • Searchable Knowledge: Don’t dump everything into the prompt (which increases costs and triggers hallucinations). Instead, build a library where agents can look up specific, high-quality “skills” when needed.
  • Short-term vs. Long-term: Use short-term memory for specific task-based context and long-term memory (via RAG—Retrieval-Augmented Generation) for architectural docs, company-wide best practices, and data catalogs.
  • Efficiency: By loading only the relevant context, you keep the model focused and cost-effective.

🌐 Event-Driven AI: Kafka & Flink

When systems grow, they need to be event-driven. Adi highlighted how companies like OpenAI use Apache Kafka and Apache Flink to manage real-time event streams that enrich AI models with live data.

  • Real-time Enrichment: Use Flink to perform summarization and analytics on events, feeding that data back into the LLM as context.
  • Automation at Scale: Imagine an agent that monitors your Jira backlog, prioritizes tickets based on product goals, and even opens Pull Requests for maintenance tasks—all triggered by event-driven architecture.
  • Open Source: Check out the Flink streaming agents API if you want to integrate these patterns into your own infrastructure.

🎯 Key Takeaways for Engineers

  1. Stop treating LLMs as stateless: Start building stateful systems that manage context across sessions.
  2. Document your “Skills”: Treat your AI workflows like a library of reusable code. If you find a good way to solve a problem, save it as a skill for the whole team.
  3. Embrace Agentic Experiences: Focus on how your software can manage the SDLC—from code quality checks to automated ticket triage.
  4. Stay Curious: The tools change every 6 months. If you haven’t revisited a technique in a while, try it again—you will likely be surprised by the improvement.

As Adi puts it: “If you can dream it, you can build it.” The ability to synthesize complex context is the most valuable skill for the modern developer. Keep experimenting, keep building, and stay ahead of the curve! 🌐✨

Listen to the full conversation with Adi Polak and Thomas Betts on the InfoQ Podcast.

Appendix