Research Highlight: Approximating Human Preferences Using a Multi-Judge Learned System

The project "Approximating Human Preferences Using a Multi-Judge Learned System" aims to bridge the gap between human and machine judgment in evaluating the quality of outputs from large language models (LLMs) by simulating human feedback through multiple rubric-based LLM judges. The research, conducted by Jose Faustino, Eitan Sprejer, Fernando Avalos, and Augusto Bernardi, involves training models like a Generalized Additive Model (GAM) and a single-layer Multi-Layer Perceptron (MLP) to predict human-like quality scores from a diverse set of LLM-generated evaluations. By using 10 distinct rubric judges to assess individual dimensions of quality and aggregating these scores, the models demonstrated improved predictive accuracy over a Naive Mean Baseline, capturing about 56–57% of the variance in human ratings. The study highlights the potential of this aggregated, interpretable approach to enhance quality assessment systems, such as Martian's model routing, by more closely approximating human preferences in a scalable manner.