July 2026 Summaries

AI budgets are increasingly scrutinized, and engineering teams are tasked with optimizing costs without impeding development. Comet addressed this by enhancing its Opik product and optimizing internal token usage, leading to the creation of Cost Intelligence, a platform that helps companies reduce AI expenses by 10–40% while maintaining performance. This initiative revealed that many tools and skills were underused, prompting a redesign of Opik's MCP server and skill libraries that significantly reduced token use and improved functionality. By analyzing AI spending with Cost Intelligence, Comet gained insights into ROI across projects, optimizing costs and performance. This approach emphasizes ongoing analysis, as AI ecosystems continually evolve, necessitating regular adjustments to maintain efficiency.

Evaluation-Driven Development (EDD) is a structured approach to AI feature development that ensures changes are effective and do not introduce regressions before they are merged into the main codebase. The process involves generating test data to simulate real-world scenarios and using an open-source tool called Opik for running experiments and evaluating the performance of new features. EDD relies on two modes of testing: a quick manual check for minor adjustments and automated experiments for larger changes, with simulated traces covering both happy paths and adversarial conditions. The evaluation process is hypothesis-driven, starting with a stated hypothesis for each feature, followed by simulations and comparisons of results using predefined metrics and judges. This method helps catch subtle errors that might not be visible in individual traces but become apparent over longer interactions, thus preventing potential costly mistakes in live environments. Alejandro Aboy, a senior data and AI engineer, exemplifies this approach in his work with Workpath, leveraging Opik to maintain alignment in enterprise strategy execution and demonstrating how offline evaluations can be more cost-effective and insightful than always-on online evaluations.