Beyond the Single Trace: How We Built Agent Diagnostics for Opik

Opik has developed a new automated agent debugging tool called Diagnostics to address the challenges of manually reading traces for AI agents in production, which often leads to inefficiencies and missed patterns. Traditional methods involve manually checking individual traces or running review sessions, but these approaches do not scale effectively and can miss silent failures. Opik's initial attempts to automate this process involved an agent that identified suspicious traces based on statistics, but this proved insufficient due to the inability to confirm findings at scale. The breakthrough came from enabling agents to query stored trace data in ClickHouse, a column-oriented database, rather than reading each trace individually. This approach allows the agent to efficiently identify and confirm widespread issues by querying the data directly, significantly improving accuracy and scalability. The ClickHouse sub-agent translates plain-language questions into SQL queries, ensuring safe and efficient database interactions. This new system effectively narrows down real patterns, confirms their impact, and remembers previously identified issues, providing a robust solution for debugging AI agents. The Diagnostics tool is now available in the free cloud version of Opik, allowing users to effectively trace and debug their own AI systems.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.