Build Your Own Judge: When a Fine-Tuned 1.7B Model Beats Jev and LLM-as-a-Judge (and When It Doesn’t)
Blog post from Comet
Using 1,000 real conversations with Opik’s Ollie assistant, the team labeled each interaction for whether the request was in scope, whether Ollie declined, and whether an attempted answer was sound, then fine-tuned a 1.7-billion-parameter model on 600 examples. On 300 held-out conversations, the model outperformed Jev and gpt-4o-mini on decline detection and scope classification, but Jev performed better at assessing factual soundness when Ollie had attempted an answer. Training took about 37 minutes, and the team tracked experiments and versioned data in Comet; an initial attempt using a general reasoning benchmark did not improve judging, underscoring the value of task-specific examples. The results suggest that small custom judges can work well for recurring interaction-pattern decisions, while factual assessments benefit from judges with broader knowledge. The findings are limited by model-generated reference labels, a single training run and a 300-example test set, and the fine-tuned model’s probabilities were poorly calibrated.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| Jev | 21 | 393 | 37 | 19 | -69% |
| AI Model Fine-tuning | 11 | 613 | 130 | 56 | -29% |
| LLM | 11 | 3,355 | 742 | 191 | -49% |
| Agent Evaluations | 6 | 112 | 50 | 27 | -33% |
| Observability | 2 | 1,980 | 454 | 150 | -36% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.