October 2026 Summaries
6 posts from Galtea
Filter
Month:
Year:
Post Summaries
Back to Blog
Customer-facing chatbots should be tested as complete, realistic conversations rather than isolated responses, with success defined by whether users safely and accurately complete tasks such as obtaining information, updating accounts, or resolving issues. Effective test suites use representative datasets drawn from support interactions, product policies, incidents, analytics, and deliberately designed edge cases, covering multi-turn context, retrieval accuracy, tool actions, handoffs to humans, safety requirements, adversarial inputs, accessibility, and language variation. The guide recommends combining deterministic checks for verifiable requirements, model-based evaluation for subjective language qualities, and human review for ambiguous or high-risk outcomes, while preserving transcripts, retrieved sources, tool traces, and state changes as evidence. Teams should establish explicit pass criteria, test changes through versioned regression suites, set category-specific release gates, and monitor live conversations for new failures, shifts in customer behavior, tool errors, and unresolved journeys. Confirmed production issues should become regression cases, allowing an initially small test suite focused on high-value customer journeys to expand continuously as the chatbot and its real-world usage evolve.
Oct 30, 2026
2,927 words in the original blog post.
Testing a voice AI agent requires evaluating the complete live-call experience rather than transcripts alone, including speech recognition, interruptions, turn-taking, reasoning, tool use, authorization, handoffs, safety, and final task outcomes. Teams should first define a clear operational contract specifying the agent’s capabilities, verification requirements, refusal boundaries, and escalation rules, then translate those requirements into realistic, evidence-backed test scenarios reflecting varied callers, audio conditions, emotions, ambiguities, system failures, and risk levels. A practical starter suite should cover normal requests, difficult-to-hear callers, corrections during interruptions, ambiguous or unsupported requests, tool errors, human handoffs, and prohibited or high-risk actions. Evaluation should combine deterministic checks for required workflow events, semantic assessments for conversational quality, and human review for nuanced or high-consequence cases, while preserving audio or approved substitutes, transcripts, traces, tool results, decisions, and version information for reproducibility. Failures should be traced to their earliest cause, incorporated into regression testing, and assessed through release gates calibrated to the workflow’s risk, with ongoing production monitoring used to identify emerging patterns and improve the test suite.
Oct 27, 2026
3,402 words in the original blog post.
Monitoring LLM quality in production requires evaluating real interactions through detailed traces that capture inputs, context, retrieval results, prompts, model versions, tool activity, outputs, costs, latency, and user feedback, rather than relying only on infrastructure metrics. Effective programs measure distinct dimensions such as task success, groundedness, relevance, safety, tool behavior, reliability, user experience, and drift, using deterministic checks for exact requirements, calibrated LLM evaluators for open-ended criteria, and human review for high-risk or ambiguous cases. Sampling should balance representative traffic with targeted coverage of failures, rare workflows, changed versions, and sensitive segments, while scorecards need clear thresholds, baselines, owners, and response procedures. When alerts arise, teams should inspect representative traces, cluster failures by root cause, reproduce confirmed issues, convert them into redacted regression tests, and compare fixes against broader evaluation sets before release. The approach also emphasizes privacy protections, including data redaction, controlled access, retention policies, and careful handling of sensitive traces, while warning against blended quality scores, uncalibrated automated judges, weak sampling, noisy alerts, and premature prompt changes.
Oct 22, 2026
2,771 words in the original blog post.
Enterprise AI evaluation platforms are presented as tools for establishing a shared quality-management process across model testing, production monitoring, human review, governance, and incident response, rather than merely generating evaluation scores. The comparison groups offerings into developer-focused frameworks such as DeepEval, Ragas, Promptfoo, and MLflow; observability platforms including Langfuse, Arize Phoenix, LangSmith, Braintrust, Galileo, Arthur, and W&B Weave; and full-lifecycle platforms such as Galtea, Maxim AI, Giskard, and Deepchecks. It recommends choosing tools according to operational needs, including code-based testing, production tracing, security red teaming, cross-functional collaboration, deployment flexibility, and evidence retention. Core enterprise criteria include versioned test suites, production scoring, non-technical review workflows, exportable evidence, and private or self-hosted deployment, while regulated sectors may also require traceability, human oversight, incident records, retention policies, and documented risk controls. The article emphasizes turning production failures into reusable regression tests and maintaining a chain of evidence from requirements through release decisions and post-release monitoring. It notes that frameworks offer flexibility for engineering teams, observability products provide production insight, and full-lifecycle tools aim to connect these functions across stakeholders, while acknowledging that the comparison is editorial, based on public information, and published by Galtea.
Oct 21, 2026
3,562 words in the original blog post.
Enterprise AI agent evaluation should assess not only final answers but also the observable workflow that produced them, including task understanding, planning, tool selection and arguments, error handling, permissions, handoffs, state management, efficiency, and final outcomes. Teams should begin with a clear task contract defining allowed capabilities, restrictions, policies, expected evidence, and consequences of failure, then test each requirement through versioned scenarios and full execution traces. The framework emphasizes that structurally valid tool calls or apparently successful answers can still conceal serious failures such as unauthorized data access, skipped verification, incorrect handoffs, unsupported claims, or actions that were never completed. It recommends deterministic checks for enforceable requirements, model-based or human evaluation for semantic and high-risk judgments, severity-based release thresholds, and negative testing for privacy, policy, prompt-injection, and recovery cases. Evaluation should operate across development, pre-release testing, and production monitoring, with results linked to system versions, reviewers, release decisions, and follow-up fixes so that failures can be traced to their first incorrect step and added to regression testing.
Oct 21, 2026
2,306 words in the original blog post.
Galtea benchmarked TypeSafe’s English-trained Jev LLM judge on 1,049 human-labelled single-turn evaluation rows translated into Spanish and Catalan, comparing its results with a tuned GPT-5.2 judge across 10 metrics. Jev’s accuracy remained essentially unchanged for Spanish inputs and declined by only 1.5 points for Catalan when its rubric questions stayed in English, but translating the questions into Spanish or Catalan reduced average accuracy further and caused a 13-point drop in toxicity detection, partly because translated wording changed the interpretation of terms such as “dismissive.” Against GPT-5.2, Jev performed significantly better on factual accuracy and non-toxicity while producing statistically similar results on seven other metrics, although both judges struggled with jailbreak resilience. The study concludes that Jev can support Spanish and Catalan input evaluation effectively when prompts remain in English, while noting that its suitability for multi-turn conversations, agent traces, and complex evaluation templates requires further research.
Oct 03, 2026
2,018 words in the original blog post.