July 2026 Summaries

ABANCA successfully deployed its AI banking assistant, SofIA, to over 2 million customers without any production incidents, significantly reducing evaluation costs and improving security measures. The deployment of SofIA, which handles queries in multiple languages, faced challenges related to manual validation processes, which were costly and limited in scope. To address these issues, ABANCA implemented Galtea's platform, which uses specification-driven test generation and calibrated LLM judges to evaluate SofIA's compliance with policy guidelines. This approach allowed ABANCA to automate the evaluation of SofIA's performance against adversarial attacks and policy compliance, leading to a 71% cost reduction per evaluation cycle and surfacing 23.6 times more vulnerabilities than manual methods. The improved evaluation process not only reduced costs but also enhanced regulatory compliance, providing a 10x return on investment and mitigating significant regulatory risks. The deployment now includes a permanent evaluation pipeline that ensures every release candidate is thoroughly tested before reaching customers, meeting the documentation requirements of the EU AI Act.

Sabadell Seguros, the insurance division of Banco Sabadell, successfully scaled its generative AI assistant, SofIA, to 6,000 sales agents by implementing a rigorous testing and evaluation program in collaboration with Galtea. This initiative was crucial in the regulated financial services sector and involved automated test generation, ensuring that SofIA's performance was on par with human agents across various scenarios, including adversarial and multi-turn conversations. The evaluation process improved key performance indicators like RAG accuracy and quality metrics by 8% and reduced refusals by 20%, providing measurable evidence of SofIA's reliability and readiness for scale-up. This systematic approach replaced guesswork with a prioritized improvement roadmap, allowing leadership to make informed decisions about deploying SofIA to millions of customers.

Automated LLM evaluation offers a method for integrating quality checks into CI/CD pipelines by running evaluations against a versioned golden dataset whenever changes are made to prompts, model versions, or retrieval configurations. This approach differs from standard test automation by employing probabilistic rather than deterministic checks and incorporating the dataset as part of the system. The process ensures that quality regressions are identified before deployment by tracking trends, managing datasets actively, and setting dynamic thresholds to distinguish between genuine regressions and false alarms. Effective implementation requires version control, consistency in evaluation settings, and structured regression tracking to support proactive quality management in AI systems. Platforms like Galtea facilitate this process by enabling comprehensive evaluation pipelines aligned with formal product specifications, enhancing the ability to maintain and improve LLM performance over time.

Offline and online evaluations serve distinct purposes in assessing the performance of large language models (LLMs), with each method identifying different issues. Offline evaluation is conducted before deployment using a fixed dataset to catch regressions caused by internal changes, effectively acting as a quality gate. However, it cannot detect external changes such as silent model updates or input drift that occur in the live system. Online evaluation, on the other hand, monitors the model's performance on real user traffic and identifies issues arising from external factors, but it is constrained by scalability and the absence of ground truth. Both evaluations are necessary for maintaining production quality, as they complement each other by addressing different failure modes. A combined approach ensures that regressions are caught before deployment while ongoing monitoring detects shifts in the live environment, preventing unnoticed degradation until users report issues. Techniques like canary evaluation and shadow scoring help transition between offline and online evaluations by testing changes on a subset of real traffic. Implementing effective sampling strategies and embedding-based drift detection further enhances the reliability of online evaluation.