Debugging our AI search assistant with agent tracing

Sentry’s Search Query Assistant translates natural-language requests into Sentry Search Syntax, and its non-deterministic outputs are evaluated against core user scenarios to identify and correct failures. While simple issues can be debugged locally through JSON outputs and mock tool calls, more complex problems involving live data, model versions, or infrastructure benefit from Sentry’s AI Conversation view, which exposes prompts, LLM generations, tool calls, timelines, and related backend traces. In one production issue, the assistant failed to return events for a custom numerical attribute because its prompt displayed the field name incorrectly rather than in the required typed search-syntax format; tracing revealed the problem in how API tool-call results were incorporated into the system prompt. After creating a reproducing evaluation, deploying a fix, and verifying corrected prompts and API parameters in production, the team also began sending local evaluation traces to Sentry, allowing developers to inspect agent behavior with the same familiar observability tools used for conventional application errors.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.