Does an AI judge need to speak your customer's language? | Galtea Blog

Galtea benchmarked TypeSafe’s English-trained Jev LLM judge on 1,049 human-labelled single-turn evaluation rows translated into Spanish and Catalan, comparing its results with a tuned GPT-5.2 judge across 10 metrics. Jev’s accuracy remained essentially unchanged for Spanish inputs and declined by only 1.5 points for Catalan when its rubric questions stayed in English, but translating the questions into Spanish or Catalan reduced average accuracy further and caused a 13-point drop in toxicity detection, partly because translated wording changed the interpretation of terms such as “dismissive.” Against GPT-5.2, Jev performed significantly better on factual accuracy and non-toxicity while producing statistically similar results on seven other metrics, although both judges struggled with jailbreak resilience. The study concludes that Jev can support Spanish and Catalan input evaluation effectively when prompts remain in English, while noting that its suitability for multi-turn conversations, agent traces, and complex evaluation templates requires further research.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.