How to build LLM-as-a-Judge evaluators that hold up in production

LLM-as-a-Judge is an evaluation framework where language models assess outputs from other models based on predefined criteria, offering benefits for scaling evaluations beyond manual review. It is critical that the evaluation criteria are clearly defined, including the target quality, inputs, allowed outputs, decision rules, and practical examples, to avoid failures common in ambiguous criteria. The framework distinguishes between code evaluators and LLM judges, where code is used for deterministic checks and LLMs for semantic evaluations. The process involves using Boolean, categorical, or ordinal labels for clarity, avoiding open numeric scores unless justified. Calibration against human labels is essential to ensure agreement with human judgment, and explanations from judges should be treated as debugging aids rather than ground truth. Evaluators should be integrated into the engineering loop, with results stored near execution records to facilitate inspection and improvement. Continuous monitoring is necessary to manage biases and drifts in the judge's performance over time, ensuring that LLM-as-a-Judge supports better engineering decisions rather than acting as a standalone metric.