September 2026 Summaries

Sentry’s Search Query Assistant translates natural-language requests into Sentry Search Syntax, and its non-deterministic outputs are evaluated against core user scenarios to identify and correct failures. While simple issues can be debugged locally through JSON outputs and mock tool calls, more complex problems involving live data, model versions, or infrastructure benefit from Sentry’s AI Conversation view, which exposes prompts, LLM generations, tool calls, timelines, and related backend traces. In one production issue, the assistant failed to return events for a custom numerical attribute because its prompt displayed the field name incorrectly rather than in the required typed search-syntax format; tracing revealed the problem in how API tool-call results were incorporated into the system prompt. After creating a reproducing evaluation, deploying a fix, and verifying corrected prompts and API parameters in production, the team also began sending local evaluation traces to Sentry, allowing developers to inspect agent behavior with the same familiar observability tools used for conventional application errors.

A developer of Cliparr, a self-hosted browser-based video clipping tool, used Sentry Application Metrics to evaluate and improve output file-size estimates without collecting users’ media or modifying self-hosted installations. By extracting the conversion engine into the public, static Cliparr Convert website and anonymously recording numeric metrics such as estimated-versus-actual size ratios, output formats, conversion quality, and completion status, the developer found that an AI-generated estimator based on resolution could underestimate files by as much as 83%. Replacing it with the standard bitrate-times-duration calculation brought estimates to within roughly 5% for transcoded videos, while the metrics also revealed that copy-mode exports were overestimated by about 16% and GIF estimates were consistently high by around 9%. The collected data showed MP4 was the most common output format and that users often converted files for compatibility rather than compression, while conversion counters showed no failures in the initial dataset. The project also instruments a progressive web app installation funnel to measure availability, prompts, acceptance, and completed installations across desktop, Android, and iOS.