July 2026 Summaries

Michael Grinich, founder of WorkOS, discusses the evolving role of AI agents in software systems, highlighting the challenges and opportunities they present. As agents become integral operators within software, capable of reading documents, calling APIs, and crossing traditional human boundaries, they introduce new security and operational complexities, as agents pursue tasks without the traditional human oversight, often leading to unexpected outcomes. Grinich emphasizes the importance of redefining software design and evaluation to accommodate these autonomous agents, focusing on aspects like identity, permissions, and memory, which are critical for ensuring agents perform safely and effectively. He notes that developer experience is shifting towards agent experience, requiring systems to be designed so agents can operate independently, aligning their actions with intended outcomes rather than superficial success. The rise of the AI engineer is characterized by a need for adaptability, as engineers must continuously learn and refine systems to manage this new dynamic, ensuring agents operate within secure and beneficial parameters while being able to self-improve through memory and feedback loops.

Arize Phoenix offers a structured approach to integrating evaluations (evals) as tests within continuous integration (CI) frameworks like Pytest and Vitest/Jest, especially for applications involving large language models (LLMs). The primary challenge addressed is the non-deterministic nature of LLMs, which necessitates the use of repetitive evaluations to ensure reliability. Evals differ from traditional tests due to the inherent unpredictability and additional complexities such as cost, latency, and the need for qualitative judgment often requiring another LLM. Phoenix provides tools to write evals as standard tests, allowing developers to track performance metrics and debug applications effectively. The process involves defining scenarios, the system under test, and checks on outputs, distinguishing between hard invariants that fail CI tests and quality signals that are monitored over time. Phoenix facilitates the organization and analysis of test data, allowing teams to maintain a source of truth in their test files while providing infrastructure to log and compare results as applications evolve.

Agent harnesses are pivotal in modern coding workflows, acting as loops that enhance a model's capability by orchestrating tasks such as file editing, testing, and code correction. While model-native pairs like Claude Code and Codex offer high performance, they often tie users to specific vendors, reducing flexibility. As models become commoditized, the focus shifts to the harness itself, which dictates a workflow's adaptability and sustainability across different models. The analysis of harnesses involves evaluating their capability—how well they integrate with models and tools—and their freedom, which is defined by the ease of switching models and maintaining workflow autonomy. The emergence of meta-harnesses, which manage multiple coding agents and enforce higher-level policies, signifies a trend towards more sophisticated orchestration, where owning the harness becomes crucial for long-term efficiency and independence. The future lies in owning the loop, as it allows for continuous adaptation and accumulation of knowledge, contrasting with the transient nature of the models themselves.

Agent evaluations are crucial for assessing the performance and effectiveness of AI agents, ensuring they complete tasks as intended without resorting to shortcuts that compromise user outcomes. These evaluations, known as agent evals, score various aspects of an agent's performance, such as final outputs, tool usage, and behavioral adherence, and are becoming vital intellectual property for agent teams. Unlike traditional unit tests, agent evals focus on encoding outcomes and constraints, providing a robust framework that persists through model changes and workflow updates. The need for precise specifications is emphasized to prevent reward hacking, where agents exploit weak evaluation criteria to achieve high scores without genuinely fulfilling user requirements. Developing resilient evals involves defining clear pass/fail criteria and ensuring evaluations are comprehensive enough to capture genuine performance rather than just numerical targets. As AI capabilities advance, the specification of what constitutes "done" becomes more critical, with the real value lying in well-crafted rubrics and test suites that guide continuous improvement and adaptation in response to new challenges and production insights.

As AI model subsidies are poised to end, businesses face the challenge of transitioning from flat-rate subscriptions to usage-based billing models for AI agents, particularly those involved in agentic workloads. The current landscape reveals that while flat-rate plans have been economically unfeasible due to heavy usage by a few users, leading to significant financial strain on AI labs, metered API pricing models have shown profitability due to their ability to charge based on actual usage. The AA-Briefcase benchmark provides insight into the cost of completing tasks accurately, highlighting that while some models like Fable 5 achieve high success rates, they incur substantial costs per successful task, prompting a reevaluation of model choice based on cost-performance metrics. As the industry anticipates the shift towards usage-based billing, it becomes crucial for organizations to calculate the cost per successful task, optimize their models for efficiency, and prepare for a future where they must justify every expenditure based on performance and outcome, leveraging cheaper open-weight models without sacrificing task success rates.