August 2025 Summaries

Claude Code and Cursor represent two distinct approaches to AI-assisted software development, catering to different user preferences and development environments. Claude Code is a CLI-first coding assistant that operates within the terminal, allowing for extensive automation and multi-file editing by executing shell commands through a lightweight large language model (LLM). It thrives in environments where developers work with tmux panes and Git branches, offering headless automation, full-repository refactors, and the ability to integrate with various tools and workflows. In contrast, Cursor is an AI-enhanced code editor built on a VS Code fork, providing a rich graphical interface focused on inline, single-file edits with support for multiple model providers. Users of Claude Code are encouraged to adopt a meticulous plan-before-code workflow, ensuring precise execution by iterating on a detailed text plan before implementation. Advanced techniques for maximizing productivity with Claude Code include parallel execution with Git worktrees, automated plan-review-execute cycles, and the use of voice control for hands-free coding. Observability tools, like Dev-Agent-Lens, help monitor the performance and efficiency of Claude Code operations, providing insights into prompt sizes and token costs.

The article introduces Dev-Agent-Lens, a tool designed to enhance observability and tracing for Claude Code, a platform known for code generation and analysis. Recognizing the limitations of conventional logs, Dev-Agent-Lens employs a proxy-based architecture using LiteLLM to centralize model routing and monitoring, while integrating OpenTelemetry and OpenInference to capture comprehensive trace data. This tool allows users to evaluate workflows for correctness, relevance, and success by examining detailed spans of model prompts, tool inputs, and outputs, thereby simplifying debugging and optimizing performance. It also helps identify cost overruns by tracking token usage, enabling users to manage budgets effectively and prevent quality degradation. The tool is versatile, supporting various real-world use cases such as security analysis, incident response, and code review, with instructions available for getting started with its implementation.

Human annotations play a crucial role in enhancing AI evaluation pipelines by providing precise feedback, despite their time-consuming and unscalable nature. The process involves collecting data, annotating it using tools like the Phoenix UI or REST API, and using these annotations to build evaluators that check the quality of AI outputs. Through iterative experiments, such as modifying models or prompts, teams can test hypotheses and measure performance changes, leading to improvements in AI systems. Even a small number of annotations can significantly ground evaluations, and tools like Arize-Phoenix facilitate the integration of annotations, evaluators, and experiments into a cohesive workflow. Custom annotation tools enable both technical and non-technical team members to contribute effectively, supporting the transition from inconsistent outputs to reliable, high-quality responses.

Handshake, an early-career network connecting students and employers, successfully deployed and scaled over 15 large language model (LLM) use cases in six months by leveraging a strategic orchestration layer and Arize AX for observability and evaluations. The company created a microservice called LLM Orca to provide a centralized and efficient pathway for launching LLM-powered features, ensuring robustness amid rapid advancements in LLM research. This service integrates with tools like Arize AX and Datadog to trace and evaluate every call, allowing for cost breakdown, automated workflows, and prompt-engineering evaluations. Since implementing Orca, Handshake has launched various LLM-powered features such as AI-generated recruiter messages and job tagging, achieving a unified quality assurance process and faster iteration with minimized risks of LLM hallucinations. Handshake's approach demonstrates how a lightweight orchestration layer and comprehensive evaluation can empower product teams to quickly ship and validate valuable features without compromising reliability or accountability.

When using large language models (LLMs) as evaluators, the inclusion of explanations and the use of chain-of-thought (CoT) prompting are crucial design choices that influence the quality and transparency of their judgments. Explanations enhance alignment with human judgments by reducing variance, exposing decision factors, and providing reusable data for retraining or improving models, while the order of explanations before or after labels has little effect on accuracy but affects the clarity of reasoning. CoT prompting, although widely adopted, shows mixed effectiveness and is most beneficial for tasks requiring complex reasoning steps, though it can increase complexity and costs in simpler tasks. Modern reasoning models, which perform internal deliberation, often outperform base models but come with trade-offs in latency and cost, making explicit CoT prompting less necessary. Therefore, explanations are recommended as part of the output to audit decisions and refine evaluation setups, with careful consideration of prompt design, score definitions, and bias mitigation strategies to ensure reliable evaluations.

The text discusses the importance and methodology of trace-level evaluations for Large Language Model (LLM) applications, as opposed to the more common span-level evaluations. While span-level assessments focus on individual steps such as tool calls or LLM responses, trace-level evaluations provide a comprehensive view of the entire workflow, assessing the success, efficiency, and relevance of the final outcome. The tutorial highlights the use of Arize AX for conducting these evaluations and provides an example through a movie recommendation agent, which utilizes multiple tools to deliver a comprehensive answer to user queries. By evaluating the entire sequence of steps, trace-level evaluations help identify whether issues arise from specific components or the overall process. This approach is particularly valuable for multi-step workflows or multi-agent systems, ensuring end-to-end reliability and relevance.

Session-level evaluations offer a comprehensive approach to assessing AI applications by focusing on multi-turn interactions rather than isolated tool calls or individual model responses, providing a holistic view of the user experience. Using the Arize Python SDK, developers can implement these evaluations by grouping traces into sessions via session IDs, which represent entire conversations, such as those between a user and a chatbot. This method allows for the analysis of session correctness, frustration, and goal achievement, offering insights into whether the AI effectively assisted the user, maintained accuracy, and prevented dissatisfaction. The process involves setting up code to attach session or user IDs to spans, preparing data for evaluation with Arize AX’s Export Client, and running evaluations using LLM-as-a-judge templates. Results can be logged back to Arize for visualization and further analysis, enabling developers to explore unsuccessful sessions, identify user frustration, and assess model performance across multiple interactions, thereby enhancing the AI system’s overall efficacy.

Arize-Phoenix provides pre-built evaluators for common scenarios, but for specialized domains like medicine, finance, and agriculture, creating a custom evaluator is often necessary to ensure high accuracy. New tutorials demonstrate how to build a custom evaluator in both Arize AX and Phoenix, starting with the creation of a benchmark dataset by annotating realistic examples and defining clear label definitions. By running experiments and iterating on the evaluation template where results disagree, users can develop a judge that aligns with their application's quality definitions. This iterative process enhances the evaluator's performance, making it adaptable to various workloads, such as validating summaries or checking citation correctness. These processes can be executed using notebooks available in both Phoenix and Arize AX platforms, with tools for configuring tracing, generating traces, and refining templates for optimal evaluator performance.

The Arize Database (ADB) is designed to support both large file and real-time event ingestion for AI data use-cases, emphasizing scalability, data ownership, and interactive feedback. ADB's architecture separates compute from data storage, leveraging Apache Parquet and Arrow for efficient data handling, allowing for elastic scaling and operational flexibility. This separation ensures that operational efficiency is maintained without compromising system resilience, enabling quick data access and minimizing costs associated with data retrieval. The use of Arrow IPC format and gRPC-based Flight technology enhances performance by reducing serialization costs, making ADB well-suited for streaming analytics. ADB also incorporates a distributed write-ahead-log (WAL) to ensure strong data delivery guarantees and effective deduplication of events in real-time data streams. Built to meet specific needs unmet by existing solutions, ADB aims to provide a robust platform for the Arize AX ecosystem, with plans for further exploration of its technical architecture and capabilities.

In July 2025, Arize AX introduced several advancements aimed at simplifying AI and agent engineering, including updates to their AI-powered assistant Alyx, which now features 30 purpose-built skills accessible through MCP for seamless integration into existing workflows. Key updates include enhanced prompt learning capabilities, session-level evaluations, and agent trajectory evaluations to improve agent performance and efficiency. The Arize Tracing Assistant was launched to provide direct support within IDEs, facilitating faster instrumentation and debugging. Additionally, OpenInference Java was released, offering comprehensive tracing solutions for AI applications, while the new Arize Database (ADB) promises high-efficiency processing of vast amounts of data. Arize also introduced a cost-tracking feature to manage AI expenses across multiple models and providers, underscoring their commitment to optimizing AI performance and usability.

A recent PwC survey reveals that 78% of enterprises are deploying AI agents or multiagent systems, with companies like Geotab, Priceline, and Booking.com demonstrating practical applications. Geotab's generative AI agent simplifies data analysis for fleet managers by translating natural language queries into SQL, enhancing decision-making without requiring in-depth SQL knowledge. Priceline introduced Penny, a real-time voice-enabled AI agent, to streamline travel bookings and improve user experience through natural interactions, boosting conversions in their e-commerce funnel. Booking.com created the AI Trip Planner, a modular system that provides personalized travel experiences by integrating internal models with large language models, resulting in improved recommendation accuracy and reduced latency. These implementations highlight the importance of custom solutions and thoughtful architecture, with each company gaining valuable insights into the optimization of AI systems and the need for robust quality control mechanisms.