What three years of watching AI in production taught us

Helicone's founders reflect on their experience monitoring AI in production over three years, observing that the key factor in AI success is not the models themselves, but the quality of the knowledge layer they rely on. Despite significant improvements in AI models from GPT-3.5 to GPT-5 and beyond, persistent problems often stem from outdated or inadequate documentation, leading to AI systems providing incorrect answers based on stale information. This has shifted the focus from prompt engineering to "context engineering," emphasizing the importance of dynamically assembling relevant and up-to-date information for AI systems. As AI models become more advanced, the bottleneck moves to how well the knowledge they access is maintained, underscoring the need for robust, structured, and accessible knowledge infrastructure. This realization led Helicone to join Mintlify, a company they were already deeply integrated with, which focuses on building the knowledge layer that AI agents use to make informed decisions. The founders highlight their alignment with Mintlify's vision of leveraging documentation not only for human consumption but as a critical component of AI decision-making processes, supported by the strong foundation and shared journey of both companies.