Why LLM Judges Disagree With Your Experts — and How to Fix It

The text discusses the discrepancy between Large Language Model (LLM) judges and subject-matter experts (SMEs) in evaluating AI-generated content, emphasizing that LLM judges, trained on Reinforcement Learning from Human Feedback (RLHF), often prioritize general helpfulness over domain-specific correctness. This structural gap is evident in sectors like finance, healthcare, and enterprise support, where adherence to regulatory and business standards is critical. The text outlines a comprehensive SME feedback workflow to address this issue, involving sampling production traces, structured annotations, and correction-note capture, which are then used to calibrate the LLM judges through few-shot refinement and prompt updates. It highlights the importance of measuring alignment between judges and SMEs using inter-rater reliability metrics like Cohen's kappa, rather than raw accuracy, to ensure the reliability of AI evaluations. The process aims to create a sustainable feedback loop that continuously improves the judge's alignment with domain-specific standards, thereby reducing the risk of production incidents and ensuring trustworthy evaluations.