Building a 100x Cheaper Trace Judge with Fireworks

As agents increasingly generate vast amounts of data, understanding their interactions with users becomes crucial, with "perceived error" serving as a valuable metric to evaluate user-agent exchanges. Perceived error focuses on instances where users believe an error has occurred, regardless of the objective accuracy of the agent's response, and is detectable through user corrections and repeated requests. A collaboration with Fireworks led to the fine-tuning of an open-source Qwen judge model to identify perceived errors across applications effectively. By leveraging data from two internal datasets, chat-langchain and Fleet, the research explored the model's ability to generalize perceived error detection across different domains. Fine-tuning demonstrated that models could achieve or exceed frontier performance while remaining cost-effective, with results showing successful transferability and performance on unseen data. The study underscores the potential for fine-tuned models to offer significant cost savings and high accuracy, emphasizing future research directions to enhance trace understanding and the development of specialized models. The refined perceived error model is set to be tested with selected customers, aiming for a broader rollout later to further refine the ability to interpret complex user-agent interactions.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.