July 2026 Summaries

In the evaluation of AI agents, offline and online methods serve distinct purposes and are often used in tandem to ensure robust performance monitoring. Offline evaluations involve testing the AI against a fixed dataset before deployment, functioning like unit tests to catch regressions and provide a controlled assessment environment. Conversely, online evaluations occur in real-time, assessing the AI's performance against live production data and user interactions, offering a dynamic and authentic measure of how the agent behaves in real-world scenarios. The combination of both methods allows teams to identify issues before release with offline evals, and to gain insights into the agent's real-time performance and user acceptance with online evals. The cost and complexity of these evaluations depend more on the scoring methods employed—such as algorithmic, signal-based, or LLM-as-judge—than on whether they are conducted online or offline.

In this detailed blog post, Linell Bonnette explores a complex issue encountered in TypeScript when using multiple middleware with the Inngest client, which causes the TypeScript types to break. The author discovers that when two middleware are applied, the return type of step.run unexpectedly collapses to an empty object due to the serialization and transformation process involving Jsonify, which composes multiple times when middleware is stacked. This issue arises from an optional property that introduces an undefined key in TypeScript’s mapped types, leading to an ill-formed type that the compiler silently accepts. Despite previous community attempts to resolve the problem, the author identifies that the root cause lies in where undefined enters the union of keys and proposes a fix by excluding undefined from the union, which effectively prevents type corruption. Further refinement is suggested by employing key remapping with 'as' to eliminate the need for additional exclusion steps, ensuring a more robust and simplified solution. This nuanced investigation highlights the subtleties of TypeScript's type system and the importance of testing type compositions thoroughly.