Presenters
Source
Navigating the AI Frontier: Building Trust and Delivering Real Value in a Rapidly Evolving Landscape 🚀✨
The world of AI buzzes with excitement, but for many established businesses, it also brings a hefty dose of skepticism and even fear. How do you integrate this powerful technology without falling into the hype trap, ensuring it solves real problems, and, crucially, earning the trust of your users?
We recently gathered a panel of brilliant product leaders to discuss this very challenge. From tackling everyday frustrations to designing for an entirely new kind of “user”—the AI agent—they shared invaluable insights on building a robust, trustworthy, and impactful AI strategy.
💡 The AI Wishlist: Solving Life’s Little Annoyances
Our discussion kicked off with a lighthearted but insightful question: What mundane problem do you wish AI could solve for you? The answers revealed a common desire for AI to simplify our complex lives:
- Debbie McMahon yearns for AI to finally regulate office temperatures, ending the eternal struggle between freezing and boiling colleagues.
- Andrew Taylor (Content Square), VP of Product, simply wants AI to make the trains work reliably.
- Bobby Gill (Tesco), Head of Product, dreams of AI managing his multiple calendars, juggling a busy life with kids, sports, and even carving out personal time.
- Karel Callens (Lusmo), Founder & CEO, wishes AI could figure out what he doesn’t know yet.
- Maya Toutountzi (Transfix by XTM), Head of Product, would love AI to keep her home tidy.
These personal wishes underscored AI’s potential to streamline even the smallest aspects of our daily existence.
🛡️ Conquering Skepticism: Quality & Transparency are King ✨
In non-native AI businesses, skepticism is a significant hurdle. Our panelists emphasized that trust is the ultimate currency, and it comes from unwavering commitment to quality and transparency.
Andrew Taylor shared Content Square’s approach with Sense Analyst, their digital experience analytics platform. Initially, it answered basic data questions. Now, it tackles complex business questions like, “how could I improve the exit rate on my PDPs?” – performing analysis comparable to their professional services team.
His secret weapon? Obsessing about quality. During beta, they implemented a thumbs up/thumbs down feedback system for every AI response. Each “thumbs down” triggered an immediate Slack notification to the team, complete with logs and traces. This allowed the team to rapidly iterate and improve system prompts. The result? Customer success stories that silenced internal skeptics, turning them into eager advocates.
Maya Toutountzi at Transfix by XTM faced similar challenges when introducing AI-generated translations 2.5 years ago. Knowing customers wouldn’t trust a “black box,” they made their AI observable. They developed a quality score that not only provided a number but analyzed why a translation performed as it did. This visibility shifted customer perception from skepticism to informed judgment. They encouraged users to start small with low-risk content and always kept a human in the loop. By learning from errors and demonstrating continuous improvement, they showed customers they controlled the output, not the LLM.
🎯 Beyond the Gimmick: Delivering Real AI Value
The power of Large Language Models (LLMs) is undeniable, but their non-deterministic nature presents a challenge. Karel Callens from Lusmo highlighted that LLMs, even with temperature adjustments, can produce inconsistent results. His solution? Don’t just slap AI on top of everything. Instead, apply core product management principles: identify customer pains, reduce friction, and increase speed.
For Lusmo, where data accuracy is paramount, they don’t let AI roam free. They confine its operational space, for instance, preventing it from freely generating SQL that could hallucinate data. Instead, they use their own query engine to ensure AI only interacts with valid data. Crucially, they remove the black box feeling by allowing end-users to always see how a chart or report was generated, trace its origins, and adapt it. This means AI handles 90% of the work, but users can always tweak it to 100% perfection, building essential customer trust.
📈 Measuring AI’s Impact: More Than Just ROI 📊
For large, non-AI native businesses like Tesco, demonstrating the Return on Investment (ROI) of AI initiatives is crucial. Bobby Gill explained that ROI is not a single metric but an ecosystem across multiple components:
- Business ROI: Focused on commercial metrics like sales, growth, and conversion. Examples include optimizing slot utilization, improving contact centers, and refining product hierarchies using AI tooling.
- Customer ROI: Delivering fast and relevant information to customers, measured by metrics like NPS score and engagement. Tesco’s personalization framework aims to drive this.
- Capabilities ROI: Building reusable, safe, and easily adopted AI capabilities across the organization.
- Workforce ROI: Making colleagues’ lives easier by automating repeatable tasks, increasing accessibility, and giving them more time back for productive, high-quality work.
Bobby also identified key pitfalls:
- Lack of a product mindset: Avoid building models just because data scientists or execs ask; focus on solving problems and delivering value.
- Data quality: Data is key. Ensure you create data for the entire organization and for AI, not just for specific products.
- Adoption: Systems only provide value if people actually use them.
Success, he concluded, hinges on strong data, clear measures, user engagement, and disciplined evaluation—transforming AI from hype to tangible value.
🤖 Building for the Future: Agents, Not Just Humans 🌐
A groundbreaking realization from the panel: in just a few years, many users won’t directly interact with platforms; AI agents will act on their behalf. This presents a fundamental shift in product design.
Maya Toutountzi noted that this means viewing your product as infrastructure for systems, not just an interface for humans. Unlike humans who can interpret messy text and explore, agents demand structured data, expected behaviors, and reliable access to tools. This forces a parallel development approach: thinking about how systems will use functionality at the same time as designing user screens. Agents operate in dynamic, non-linear workflows, requiring composable functionality that can be used independently. Observability also becomes paramount, giving customers control over these complex, agent-driven processes.
Andrew Taylor echoed this, highlighting Content Square’s investment in MCP (Multi-Cloud Platform) for data delivery to agents. He outlined practical considerations:
- Volume: Agents can generate 100,000 requests per day, drastically different from human usage (10-100). This demands careful pricing and scaling strategies.
- Skills: Instead of giving agents all data, create specific skills with relevant data points for particular jobs, which also helps control API call costs.
- Feedback: While humans provide feedback, agents currently don’t. A human in the loop is essential for short-term feedback, but Andrew anticipates agents developing feedback capabilities within 6-12 months, again requiring an obsession with quality.
He sees two huge opportunities for B2B SaaS: serving customers’ agents and fostering an ecosystem of talking agents built by tech partners. This era promises rapid development and unprecedented sophistication.
🚀 Navigating Speed & Risk: The “Move Fast, Break What?” Era 🛠️
The classic tech mantra “move fast and break things” faces new scrutiny with AI. While Karel Callens believes the mantra still holds, he emphasizes that what you’re allowed to break has changed. His team engages in more experimentation, alpha, and beta releases with feature flags than ever before, accelerating innovation. However, when something touches a customer workflow, they add guardrails, security, permission levels, and human intervention.
A key enabler for this balance is becoming more modular and composable, allowing easy swapping of components internally and externally. For instance, they never code for just one model, using abstraction layers like LangChain to swap models seamlessly. They also empower customers with drag-and-drop capabilities for charts, allowing them to swap in proprietary components. Karel strongly advocates for Test-Driven Development (TDD), asserting there’s “no excuse” for engineers not to use it, as it easily prevents regression bugs while maintaining speed.
Bobby Gill reinforced the importance of a robust innovation framework to manage speed without chaos. He advised focusing on building lasting components and not over-engineering rapidly changing parts. Crucially, he stressed the importance of launching products at the right time for your business, rather than sprinting to keep up with others.
The Future of AI: Thoughtful Innovation and Unwavering Trust
The panel’s collective wisdom paints a clear picture: AI is a transformative force, but its successful integration demands more than just technical prowess. It requires a deep understanding of user needs, an unwavering commitment to quality, transparent practices that build trust, and a forward-thinking approach to product design that anticipates the rise of AI agents. By embracing these principles, businesses can confidently navigate the AI frontier, delivering real value and shaping a more intelligent future.