July 2026 Summaries

Digibee, founded nine years ago to address integration issues in enterprise tech stacks, has evolved its platform to incorporate AI agents that function as digital integration engineers. The reimagined platform allows users to specify integration requirements interactively, with AI-driven Digital Workers generating detailed specifications and production-grade workflows in significantly less time. The integration process, which traditionally required weeks, now takes hours, and the first draft is often over 90% accurate. The implementation of this AI-centric approach necessitated the use of Opik, a prompt management platform, which streamlined the versioning of numerous prompts without full deployment cycles, saving months of engineering time. Opik also facilitates the evaluation and refinement of prompts through simulated sessions, ensuring that the generated workflows align closely with human-built pipelines. This strategic shift aims to replicate the original platform's success in reducing integration time by 10-20 times, with ongoing enhancements planned for the coming months.

Opik has developed a new automated agent debugging tool called Diagnostics to address the challenges of manually reading traces for AI agents in production, which often leads to inefficiencies and missed patterns. Traditional methods involve manually checking individual traces or running review sessions, but these approaches do not scale effectively and can miss silent failures. Opik's initial attempts to automate this process involved an agent that identified suspicious traces based on statistics, but this proved insufficient due to the inability to confirm findings at scale. The breakthrough came from enabling agents to query stored trace data in ClickHouse, a column-oriented database, rather than reading each trace individually. This approach allows the agent to efficiently identify and confirm widespread issues by querying the data directly, significantly improving accuracy and scalability. The ClickHouse sub-agent translates plain-language questions into SQL queries, ensuring safe and efficient database interactions. This new system effectively narrows down real patterns, confirms their impact, and remembers previously identified issues, providing a robust solution for debugging AI agents. The Diagnostics tool is now available in the free cloud version of Opik, allowing users to effectively trace and debug their own AI systems.

An agent harness is a critical component in the deployment of LLM-powered features, serving as a structured framework that integrates software systems, tool calls, and evaluation functions to optimize the performance and reliability of AI agents. It addresses common production failures by providing context management, state persistence, tool orchestration, and safety controls, thereby enhancing the model's ability to handle complex tasks and interactions with external systems. The harness ensures that AI agents do not lose track of goals or repeat errors by implementing strategies like compaction, selective context injection, and validation of tool calls. Unlike an agent framework, which offers building blocks for developers, an agent harness is a runtime environment that includes pre-configured components for managing execution and error recovery, shifting the focus from component assembly to system optimization. This distinction emphasizes the harness as the primary area for engineering innovation, where observability and performance metrics are crucial for maintaining the reliability and efficiency of AI systems in real-world applications.

Facing constraints on AI token expenditure, an engineering team optimized their system both internally and through their product, Opik, to manage costs without sacrificing performance. They tackled the problem by overhauling their MCP server, which initially exposed thirty tools per REST endpoint, complicating and slowing down the process as the system had to navigate through each tool's schema. By consolidating their operations into just four versatile tools, they improved efficiency and reduced token waste, allowing the model to learn a unified pattern for all data management tasks. This restructuring was guided by three key decisions: making tools universal with entities as parameters, adopting a self-healing interface that corrects mistakes on-the-fly, and ensuring every token's necessity. These changes saved approximately 30% on token costs by managing context and configurations effectively. The team also developed Cost Intelligence, a tool that identifies and rectifies inefficient coding agent configurations, offering insights into spend management and helping to maintain an optimal environment as the platform evolves.

AI budgets are increasingly scrutinized, and engineering teams are tasked with optimizing costs without impeding development. Comet addressed this by enhancing its Opik product and optimizing internal token usage, leading to the creation of Cost Intelligence, a platform that helps companies reduce AI expenses by 10–40% while maintaining performance. This initiative revealed that many tools and skills were underused, prompting a redesign of Opik's MCP server and skill libraries that significantly reduced token use and improved functionality. By analyzing AI spending with Cost Intelligence, Comet gained insights into ROI across projects, optimizing costs and performance. This approach emphasizes ongoing analysis, as AI ecosystems continually evolve, necessitating regular adjustments to maintain efficiency.

Evaluation-Driven Development (EDD) is a structured approach to AI feature development that ensures changes are effective and do not introduce regressions before they are merged into the main codebase. The process involves generating test data to simulate real-world scenarios and using an open-source tool called Opik for running experiments and evaluating the performance of new features. EDD relies on two modes of testing: a quick manual check for minor adjustments and automated experiments for larger changes, with simulated traces covering both happy paths and adversarial conditions. The evaluation process is hypothesis-driven, starting with a stated hypothesis for each feature, followed by simulations and comparisons of results using predefined metrics and judges. This method helps catch subtle errors that might not be visible in individual traces but become apparent over longer interactions, thus preventing potential costly mistakes in live environments. Alejandro Aboy, a senior data and AI engineer, exemplifies this approach in his work with Workpath, leveraging Opik to maintain alignment in enterprise strategy execution and demonstrating how offline evaluations can be more cost-effective and insightful than always-on online evaluations.