AI agent analytics: why your click data can’t see the whole picture

Kylan Gibbs, co-founder and CEO of Inworld AI, emphasizes the need for a new approach to measuring AI product success, highlighting the limitations of traditional analytics methods designed for button-based user interactions. He points out the "black box problem," where AI-driven conversations lack visibility, making it difficult to track and optimize user engagement in AI-native products. Gibbs advocates for treating various elements of AI systems, such as models, prompts, voice, and tools, as testable variables to improve performance and align metrics with real user behavior. He shares examples of companies that succeeded by experimenting with different AI configurations, demonstrating significant cost reductions and enhanced user engagement. Gibbs advises teams to focus on one variable at a time, connecting it to key business metrics, to build a robust measurement infrastructure that provides a competitive edge in the evolving AI landscape.