May 2026 Summaries

In 2024, prompt caching revolutionized the economics of AI agent sessions, challenging the traditional belief that smaller prompts are cheaper by introducing a system where larger, stable prompts can be cached and reused at a significantly reduced cost. This shift means that loading a large amount of context initially and reusing it for subsequent interactions can be more cost-effective than constantly sending smaller prompts. The article illustrates this with data from real production sessions, showing that agents utilizing high cache hit rates incur lower costs despite larger token counts, inverting previous optimization strategies that focused on minimizing prompt size. The article emphasizes the importance of adapting to this new caching paradigm, as traditional practices now lead to inefficiencies and higher costs, particularly when evaluating AI models, where full prompt scanning negates any caching benefits. Consequently, engineers are encouraged to recalibrate their approach, focusing on maximizing cache utilization rather than minimizing prompt size, to develop more efficient and capable AI agents.

The text discusses the financial and operational challenges associated with using large language models (LLMs) for evaluating AI agents at production scale, highlighting the high costs and potential inaccuracies when using frontier models like GPT-4.1. It explores the limitations of cheaper alternatives, such as switching to less expensive models or sampling, which can lead to blind spots in detecting rare but critical failures. The solution proposed is using small language models (SLMs) that are more cost-effective and can maintain accuracy when fine-tuned on specific domain data. Luna Studio is introduced as a turnkey solution for training custom SLM judges, allowing companies to use a small set of labeled examples to create effective evaluators without extensive engineering projects, thus addressing the issues of data scarcity, scaling, and accuracy in evaluating AI agents.

Eval Engineer is a skill bundle designed for Claude Code and OpenAI Codex that aims to streamline the process of diagnosing and fixing issues in coding agents by integrating with Galileo, a system that provides valuable evidence from production environments. By examining logs, metrics, and traces, Eval Engineer can identify the root cause of a problem, propose a bounded fix, and create a verification plan to ensure that the solution is effective. It is not a replacement for existing systems but rather complements them by making the evaluation lifecycle part of the development workflow. This tool is particularly useful for AI engineers, researchers, field debugging engineers, and site reliability engineers, as it allows them to work within their existing environments while providing clear and verifiable artifacts for every diagnosis and fix plan. Eval Engineer is open-source and customizable, allowing teams to adapt it to their specific needs by configuring what evidence is relevant, which files can be edited, and what verification commands should be used. The tool emphasizes small, reviewable changes and encourages human oversight in product decisions, ensuring that automation does not replace expert judgment.

Cursor is an AI-native integrated development environment (IDE) equipped with a built-in agent that can manage GitHub repositories through natural language prompts, offering functionalities like creating branches and merging pull requests. However, its current allowlist system, which controls what the agent can access, is only applied individually per developer, leading to a lack of centralized governance and potential operational issues. To address this, the implementation of Cursor hooks allows external command execution at specific points in the agent loop, enabling centralized policy enforcement via the Agent Control server. This approach involves using hooks to intercept agent calls and evaluate them against centrally configured rules, which dictate permissible operations, thus ensuring organizational policy consistency across all developers. By committing the hook script and configuration directly to the repository, organizations can maintain a team-wide policy, reducing configuration drift and onboarding errors. Additionally, Eval Engineer complements this system by enabling post-execution evaluation to understand agent behavior, thereby facilitating continuous improvement and operational efficiency.

The text discusses the limitations of using static golden datasets for evaluating AI models, especially in dynamic real-world environments, and proposes a shift towards continuous evaluation methods. It highlights how static datasets, while providing stable performance benchmarks, often fail to capture the evolving nature of production inputs, leading to a mismatch between reported and actual performance. This gap is exacerbated by distribution drift and the introduction of new user behaviors or system changes. The text suggests that dynamic evaluation, which involves sampling live production traffic and routing uncertain cases to subject matter experts for annotation, can help close this gap by adapting to real-world conditions. This approach not only improves evaluator models with each annotation cycle but also ensures that performance metrics remain aligned with actual usage patterns. The text further explains how platforms like Galileo facilitate this process by providing tools for continuous learning feedback, structured annotation workflows, and real-time metrics to maintain accuracy and relevance in AI system evaluations.

AI compliance presents a significant challenge for engineering leaders, requiring a balance between rapid deployment and regulatory adherence. The key to reconciling these seemingly conflicting priorities lies in embedding compliance into the development workflow from the outset, rather than treating it as a separate, post-development audit process. By incorporating automated compliance checks into CI/CD pipelines, engineering teams can catch and address potential compliance issues early in the development cycle, thereby reducing rework and accelerating time-to-market. This approach, known as compliance-by-design, shifts the focus from traditional compliance frameworks to a more integrated model that treats compliance as a continuous practice, producing ongoing evidence through tools like Galileo's Metrics Engine and Runtime Protection. These tools automate the evaluation of AI systems against compliance criteria, ensuring regulatory requirements are met without impeding innovation. The strategic use of risk-tiering allows teams to allocate resources proportionally, focusing on high-risk areas while maintaining efficiency in lower-risk workflows. This shift in mindset transforms compliance from a hindrance into a catalyst for faster, more reliable AI development.

The text discusses the challenges of ensuring reliability in AI systems, particularly in distinguishing between non-determinism and brittleness. Non-determinism is when identical inputs produce different outputs due to factors like stochastic sampling, which can be controlled with engineering techniques such as temperature settings and fixed seeds. Brittleness, however, arises when semantically equivalent inputs yield different outputs due to slight variations in phrasing, which is not addressed by controlling temperature. The text highlights the importance of identifying brittleness through methods like paraphrase testing and adversarial input variation, as well as the cost implications of conflating these issues. It emphasizes that production-ready AI must demonstrate stable behavior across a range of real-world input variations, rather than just achieving high accuracy on clean test sets, to avoid costly errors and ensure reliability.

The text discusses the importance of distinguishing between Human-in-the-Loop (HITL) and Expert-in-the-Loop (EITL) methodologies in AI systems, particularly in high-stakes domains like healthcare, legal, and financial services. HITL is a runtime control mechanism where humans make decisions on specific production-agent actions, ensuring safety and compliance. In contrast, EITL focuses on the credibility of evaluation systems, using domain experts to define, calibrate, and refine metrics that grade AI output. The challenge lies in closing the agreement gap between automated judges and subject matter experts (SMEs) to ensure eval systems are reliable enough for release decisions without constant expert oversight. The text also outlines strategies for building and calibrating expert evaluation panels, emphasizing the importance of structured annotation, rubric design, and sampling strategies to maintain measurement credibility. By transforming expert feedback into automated judges, organizations can achieve scalable, trustworthy evaluations that support both real-time decisions and audit readiness.

An LLM judge, or large language model judge, requires continuous calibration rather than a one-time setup to maintain its accuracy and relevance in dynamic production environments. Over time, factors such as model updates, prompt drift, and domain shifts can lead to judge drift, where the model's evaluations no longer align with human expert judgments, resulting in compounding errors in downstream applications. To counteract this, the process of calibration involves using stratified sampling to capture a representative range of outputs, engaging subject matter experts (SMEs) to provide feedback and corrections, and updating anchor examples and rubrics based on this input. Inter-rater reliability (IRR) metrics, like Cohen's kappa, are crucial for assessing the degree of alignment between the judge and human reviewers, as they account for chance agreement and provide a more accurate measure of judge performance than raw agreement rates. Continuous calibration, therefore, involves a systematic loop of sampling, feedback collection, anchor updating, and validation to ensure the judge remains aligned with human expectations and adapts to evolving production demands.

As AI regulation intensifies, particularly with the EU AI Act's phased enforcement through 2026, engineering teams face the challenge of designing systems that comply without constant rewrites. The text outlines the need for adaptable architecture that decouples compliance policies from application code, centralizes control, and employs continuous evaluation for audit-grade evidence. AI governance has evolved into a board-level responsibility marked by sanctions for failures, demanding transparency, accountability, and runtime control across industries such as healthcare, finance, and legal. The EU AI Act's extraterritorial reach necessitates compliance for any AI systems interacting with EU markets, urging teams to prepare through architectural readiness, continuous evals, governance integration, and a future-proofing checklist to mitigate architectural debt as regulations evolve.

A recent exploration into evaluation methodologies for customer support highlights the limitations of fixed rubrics and the potential benefits of instance-specific rubrics, which adapt evaluation criteria to each unique input. Traditional fixed rubrics, often based on generic dimensions like helpfulness and coherence, can produce misleadingly high scores even as real-world quality issues remain undetected, especially in heterogeneous and high-stakes environments. The proposed instance-specific approach involves a three-step process that analyzes each input, generates tailored evaluation criteria, and scores outputs accordingly, leading to more accurate assessments in contexts such as diverse customer support queries or multi-step autonomous-agent workflows. While this method can enhance interpretability and governance by providing detailed criterion-level feedback, it also involves higher computational costs and potential consistency challenges, necessitating a hybrid strategy that combines fixed rubrics for volume tasks with instance-specific evaluations for complex cases. The integration of subject matter expert (SME) annotations ensures the reliability of generated criteria, making this approach particularly suitable for regulated industries or tasks with significant within-domain variation.

Tool Misuse and Exploitation (OWASP ASI02) is a significant threat in agentic AI applications, where agents misuse legitimate tools due to factors like prompt injection or misalignment. This phenomenon can lead to resource overload, data exfiltration, and unauthorized actions, as seen in cases where AI agents use tools in unintended ways without exceeding their permissions. The risk is exacerbated in multi-agent architectures where agents autonomously decide which tools to use, potentially propagating malicious instructions across agents. The OWASP framework outlines several attack patterns, including tool poisoning, indirect injection, and over-privileged API access, which can be mitigated through strategies like adaptive tool budgeting, semantic validation, and robust policy enforcement. Centralized policy enforcement systems like Galileo's Agent Control provide defense by updating policies across all agents in real-time, ensuring that agents operate within defined security constraints while maintaining observability and audit trails.

Langfuse, an open-source platform known for its flexibility and self-hosting options, offers LLM observability and prompt management but faces limitations when scaling from prototypes to production, particularly due to its lack of runtime intervention and reliance on costly LLM-as-judge evaluations. While it appeals to developers seeking vendor independence, Langfuse's operational risks and complexity, exacerbated by its multi-component stack, lead many teams to explore alternatives. Among these, Galileo stands out by integrating observability, evaluation, and runtime protection in one platform, offering real-time output blocking, proprietary eval models, and self-service metric creation without engineering reliance. Other alternatives like LangSmith and Arize AI cater to specific ecosystems but may lock users into particular frameworks or lack comprehensive runtime controls. The demand for platforms that can offer real-time intervention, cost-effective evaluation, and flexible deployment options is increasingly driven by the need for reliable production-grade control that Langfuse struggles to provide.

Agent governance is a set of structures, policies, controls, and oversight mechanisms designed to manage the delegated authority of autonomous AI systems, particularly those capable of making real-time decisions and taking actions with real-world consequences. Unlike traditional AI governance, which focuses on models, data pipelines, and pre-deployment risk assessments, agent governance extends to runtime behavior, tool access, and continuous operational oversight. This approach addresses the limitations of model-level governance by implementing centralized, hot-reloadable policies to replace hardcoded guardrails, ensuring that autonomous agents operate within defined boundaries. Risk classification and compliance alignment are used to integrate these systems into existing governance frameworks, while observability and runtime intervention tools are employed to detect and mitigate policy violations and unsafe actions. As the use of AI agents in enterprise environments grows, strong agent governance becomes crucial to prevent cascading failures, unauthorized transactions, and regulatory penalties, while enabling scalable and trusted automation.

Production LLMs are prone to failure rates of 5% to 30% due to their non-deterministic outputs, with state-of-the-art models still experiencing hallucinations in 15–20% of responses, as noted by a ResearchGate review. Without dedicated reliability infrastructure, teams often face challenges in debugging and preventing unsafe outputs from reaching users. LLM reliability platforms address this by integrating observability, evaluation, and runtime protection to form a systematic defense layer. These platforms collect telemetry data, apply automated quality assessments, and use interventions to prevent failures, distinguishing them from traditional monitoring tools. Notable platforms like Galileo, LangSmith, and Arize AI offer various capabilities, such as distributed tracing, eval models, and runtime protection, tailored to different infrastructural needs and compliance requirements. While some platforms focus on open-source solutions offering data sovereignty, others provide proprietary models to reduce costs and enhance reliability. The most effective strategy involves layering these tools to create a comprehensive lifecycle platform that not only observes failures but actively prevents them, thereby ensuring consistent performance in production environments.

With the rapid integration of AI agents into enterprise applications, low-latency large language model (LLM) evaluation tools have become essential for maintaining production quality control and addressing the challenge of evaluating model outputs quickly enough to prevent hallucinated or unsafe responses from reaching end users. Traditional LLM-as-judge evaluations are too slow for inline use, prompting the need for tools capable of millisecond-scale evaluation, such as Galileo's Luna-2, which offers sub-200ms latency and transforms offline evaluations into real-time production guardrails. These tools measure various metrics like hallucination detection and instruction adherence and allow for synchronous evaluation within the request lifecycle, enabling real-time intervention. While some tools, like LangSmith and TruLens, focus on development-time debugging and offline analysis, others, like Lakera and Guardrails AI, emphasize security and schema enforcement. Companies must balance using open-source frameworks for development testing and commercial platforms for inline production evaluation to ensure both development-time testing and real-time runtime evaluation.