Content Deep Dive

Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn’t)

Blog post from Comet

Post Details
Company
Date Published
Author
Aswin Thiyagarajan
Word Count
2,549
Company Posts That Month
5
Language
English
Hacker News Points
-
Post removed?
No
Summary

Using 1,000 real conversations with Opik’s Ollie assistant, the team labeled each interaction for whether the request was in scope, whether Ollie declined, and whether an attempted answer was sound, then fine-tuned a 1.7-billion-parameter model on 600 examples. On 300 held-out conversations, the model outperformed Jev and gpt-4o-mini on decline detection and scope classification, but Jev performed better at assessing factual soundness when Ollie had attempted an answer. Training took about 37 minutes, and the team tracked experiments and versioned data in Comet; an initial attempt using a general reasoning benchmark did not improve judging, underscoring the value of task-specific examples. The results suggest that small custom judges can work well for recurring interaction-pattern decisions, while factual assessments benefit from judges with broader knowledge. The findings are limited by model-generated reference labels, a single training run and a 300-example test set, and the fine-tuned model’s probabilities were poorly calibrated.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Jev 21 393 37 19 -69%
AI Model Fine-tuning 11 613 130 56 -29%
LLM 11 3,355 742 191 -49%
Agent Evaluations 6 112 50 27 -33%
Observability 2 1,980 454 150 -36%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.