What is Evals Engineering?

Evals engineering is a discipline focused on creating evaluation processes to measure the effectiveness and reliability of Generative AI (GenAI) systems, which traditional machine learning evaluation metrics cannot adequately assess. It emphasizes the need for continuous evaluation throughout the development and production stages to catch quality issues such as hallucination rates, context adherence, and response quality before they reach users. Unlike traditional software testing, evals engineering requires monitoring of GenAI systems in real-time and involves metrics like context adherence, correctness, and toxicity, as well as practices like automated scoring, feedback loops, and production monitoring to ensure systems remain reliable over time. Galileo is presented as a comprehensive solution, offering automated evaluations, real-time monitoring, and intelligent failure detection to enhance the scalability and trustworthiness of GenAI systems.