Tracing. Where Jev Fits

Paper Compute evaluated TypeSafe’s Jev as an experimental automatic labeler for agent-session traces, comparing it with GPT-4o mini, Claude Haiku 4.5, and locally run Qwen3 models on 1,781 engineer turns from 138 private team sessions. The system asked whether engineers pushed back on agents, asked questions about their work, stated lasting preferences, or supplied information that would have changed an earlier action, producing candidate labels such as pushback, question, observation, and missing knowledge. Jev had the fastest median response time at 188 ms, compared with roughly 1,023–1,050 ms for the hosted general-purpose models and 2,311 ms for Qwen3 8B, while recovering all 19 human-accepted question-label examples in a 35-turn reviewed subset; however, it also flagged six rejected examples, so its output is intended as a human-review queue rather than a final verdict. Estimated API costs for similar workloads favored Jev over GPT-4o mini and Claude Haiku 4.5, though local hardware, energy, and review costs were excluded. The author emphasizes that the reviewed examples informed question design, making the results development-set agreement rather than a measure of generalizable accuracy, and describes how span-level labels and detailed traces could help teams identify useful human corrections, inspect their context, and potentially train future agent-review systems.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.