How to Use Total Variance in LLM Evals

Total variance is a critical measure in evaluating Large Language Models (LLMs), helping discern the stability and reliability of evaluation scores across various test cases, repeated runs, model calls, and judge decisions. This metric is essential for LLM applications, such as agents or prompt chains, where a single average score may not suffice due to inherent variability in model outputs. Total variance comprises between-test-case variance, within-test-case variance, judge variance, and system variance, and it's instrumental in determining how much of the evaluation result is signal versus noise. Effective use of total variance involves conducting repeated runs, maintaining clean dataset separations, and focusing on per-test-case variance to identify unstable examples. Debugging efforts are enhanced by starting with high-variance cases, using traces to understand the causes of variability, and ensuring consistent scoring methods. Reporting should include detailed metrics like mean scores, total observed variance, and judge configuration to guide engineering actions. This approach helps separate genuine prompt improvements from noisy evaluation results, providing a structured framework for debugging and improving model performance in production environments.