March 2026 Summaries

NVIDIA CEO Jensen Huang envisions a future where companies operate with approximately 100 AI agents per employee, a vision already materializing as McKinsey reports 25,000 AI agents working alongside 60,000 human employees. This shift signifies that AI agents are becoming integral operational infrastructure rather than mere concepts, prompting enterprises to rethink governance strategies. The current challenge lies in bridging the gap between established governance policies and real-time agent activities, as traditional tools often fail to trace AI agents’ decision-making processes accurately. Misconfigurations, misunderstood contexts, and behavioral drifts pose significant risks, emphasizing the need for enhanced runtime visibility and enforcement. Organizations must focus on creating comprehensive audit trails and implementing observability-driven sandboxing to preemptively manage potential errors and compliance issues. As regulatory frameworks like the EU AI Act evolve, the importance of AI governance infrastructure becomes paramount, with platforms such as Arize offering solutions to trace, evaluate, and govern AI agents effectively in production environments.

Published on March 19, 2026, the article "Managing Memory in AI Agents: Beyond the Context Window" explores the strategies employed in developing Alyx 2.0, an AI engineering agent, to handle the extensive data generated during its operation. The article, co-authored by Chris Cooning, Priyan Jindal, Sally-Ann DeLucia, and Jack Zhou, delves into the limitations of context windows and the necessity of efficient data management for the agent's functionality. Key strategies include middle truncation with IDs to preserve meaningful data, an emulated file system in memory, deduplication and message hygiene to reduce redundancy, and the use of sub-agents to manage high-volume data tasks separately. The article also highlights the challenges faced, such as the failure of using LLM-based summarization for context compression, and the ongoing efforts to refine context management through improved heuristics, tooling, and sub-agent patterns. This publication is part of a series on the development of Alyx, with upcoming installments focusing on testing and evaluation methods for non-deterministic systems.

Banks adopt the Arize ecosystem due to its compatibility with their federated organizational structures and stringent regulatory environments. Large banks often operate as decentralized systems with individual business units managing their own technology stacks, creating challenges for centralized AI platforms. The Arize ecosystem, including Arize Phoenix and Arize AX, provides flexible deployment options that accommodate varying levels of AI maturity and independent infrastructure decisions within banks. It allows for seamless scaling from individual units to enterprise-wide deployments, supporting both technical and non-technical stakeholders involved in governance and risk management. Arize enhances observability, compliance, and security by offering tools that align with regulatory requirements, such as auditability, evaluation workflows, and robust access controls, while also facilitating collaboration across diverse teams. This adaptability and support for operational complexity make Arize a preferred choice for banks looking to implement AI systems effectively within their unique environments.

Arize AX has announced the integration of native support for NVIDIA NIM as a dedicated AI model provider, allowing enterprises using NIM-deployed models to connect directly to the Arize platform for experimentation, production monitoring, and online evaluations. NVIDIA NIM microservices, part of NVIDIA AI Enterprise, facilitate the secure and reliable deployment of high-performance AI model inferencing across various infrastructures and support a wide range of model families. This integration allows for seamless access to NVIDIA models via Arize, enhancing the ability to assess and improve model performance continuously in production environments. It particularly benefits enterprises with specific data residency or compliance needs by providing a self-hosted deployment option. Arize's integration with NVIDIA's ecosystem, including NVIDIA NeMo, strengthens the collaboration by enabling a comprehensive, connected workflow that includes deploying models, observing production traffic, curating datasets, and fine-tuning improved models. This development aims to enhance the efficiency and reliability of AI systems for enterprises by providing full visibility and continuous evaluation, supporting the growing trend of on-premises AI inference driven by compliance and data residency requirements.

Arize Skills introduces a new era in AI engineering by enabling coding agents to autonomously execute complex tasks without needing constant user input or specific syntax instructions. With the launch of Arize Skills, users can now install pre-built instruction sets that provide agents with native knowledge of Arize workflows, allowing them to export traces, add instrumentation, manage datasets, run experiments, and optimize prompts seamlessly. These skills are compatible with various coding agents, including Cursor, Claude Code, Codex, and others, making it easier for agents to perform tasks such as generating deep links in the Arize UI and analyzing codebases to implement tracing. In practice, this functionality was demonstrated by having Claude Code build a financial agent, showcasing how Arize Skills can automate complex workflows, such as setting up tracing for financial applications and evaluating the effectiveness of financial advice provided by AI. This advancement, alongside previous releases like Alyx 2.0 and the AX CLI, embeds Arize directly into engineering workflows, significantly enhancing the autonomy and efficiency of AI software development.

An AI-powered tool was developed to generate newsletters from recent tweets by using Claude, but initial results showed good writing with flawed details, such as incorrect URLs and missing links. To address these issues, a coding agent was employed to iteratively improve the process by running evaluations, fixing errors, and suggesting next steps autonomously. The improvement process revealed that data preprocessing, rather than prompt engineering, was more effective in enhancing output quality. Evaluators were used to measure dimensions such as faithfulness, structure adherence, and link accuracy, leading to enhancements in data handling and prompt instructions. However, human judgment was necessary to redefine what the evaluations should measure, leading to the introduction of a new evaluator focusing on content coverage rather than link completeness. This iterative approach highlighted the importance of accurate evaluations and human decision-making in guiding AI agents, demonstrating that while agents are efficient in optimizing tasks, humans are essential in setting objectives and ensuring meaningful outcomes. The entire project, including the code and evaluation suite, is available as open-source for further experimentation.

In 2025, the anticipated rise of agents fell short due to their inability to handle complex workflows effectively, which was attributed to a lack of planning rather than capability. The article discusses how the integration of structured planning tools into Alyx 2.0 transformed it from a basic task executor into a sophisticated workflow orchestrator. By implementing a system where planning is treated as a structured tool call rather than a prompt instruction, Alyx was able to maintain focus on multi-step tasks, manage its progress through four task statuses (pending, in_progress, completed, blocked), and ensure completion before finishing any task. The article highlights the importance of maintaining the plan's visibility throughout interactions, with the plan being consistently updated and positioned directly after the system prompt. This structured approach not only improved task completion rates but also unlocked emergent capabilities such as the ability to autonomously chain prompts together. The authors emphasize that this planning framework, which is adaptable and enforces completion through tool-level validation, is crucial for developing agents capable of handling complex workflows.

Alyx 2.0, the AI engineering agent for Arize AX, has advanced from a user interface to a command-line interface (CLI), allowing for seamless integration with coding agents through natural language intents and machine-readable data. This transition enables users to perform tasks such as analyzing common user questions or identifying failing tool calls directly in their code editors by leveraging local files and various coding agents like Cursor or Codex. The developer preview of the AX CLI supports structured output formats such as JSON, CSV, and Parquet, facilitating automation and reducing the need for manual syntax writing. Future developments aim to introduce headless debugging capabilities and higher-level workflows to further enhance user experience and efficiency.

In evaluating tool-calling agents, the introduction of Large Language Models (LLMs) to tools introduces potential points of failure, such as incorrect tool selection or improper tool invocation, necessitating distinct measurement and correction methods. Phoenix provides a framework to assess these issues through two prebuilt evaluators: tool selection and tool invocation, which function without labeled datasets by reasoning from conversational context. In a travel assistant demo, Phoenix's evaluation workflow identifies and iterates on failures, such as incorrect date usage and semantic interpretation issues, by customizing evaluators to align with specific domain requirements. This iterative process not only improves the assistant's performance but also calibrates evaluators to ensure they accurately reflect the intended tool-calling behavior, with the results highlighting the importance of adapting evaluation tools to meet specific use-case constraints.