July 2026 Summaries

In July 2026, Inngest encountered four significant service incidents that disrupted its operations and impacted its customers. On July 10, a connectivity issue with a third-party provider led to a fallback to default configurations, degrading function execution and run scheduling, which was later resolved by rolling back the release. On July 16, two incidents occurred: the first increased execution latency and checkpointing errors due to a startup dependency issue in an upgrade-readiness state store path, and the second involved a failure to publish execution metrics because of configuration drift, both of which were addressed by disabling the problematic paths and restoring permissions. Finally, on July 23, a capacity-allocation failure in a new storage backend halted event processing and delayed function execution, resolved by isolating the affected topic and scaling services. Inngest has taken corrective measures, including adding monitoring and alerting systems, to prevent future occurrences and minimize customer impact.

In the rapidly evolving field of AI agent architecture, the emphasis is shifting from selecting the right patterns to creating adaptable, durable execution layers that withstand constant change. While the context and compute layers, which encompass knowledge and code execution environments, are expected to change frequently, the execution layer—responsible for orchestrating tasks, handling retries, and maintaining state—should be stable and robust to facilitate seamless transitions between patterns. The key to successful AI engineering lies in decoupling these layers, thus avoiding the pitfalls of coupling, which can lead to extensive rewrites each time a new pattern or capability emerges. By focusing on modular and resilient execution mechanisms, teams can ensure their architectures are agile, allowing for quick adaptation to new advancements without significant overhaul. This approach not only supports current needs but also prepares for emerging architectures that require long-running, autonomously operating agents capable of dynamic coordination and evaluation.

The article explores the challenges of ensuring that an AI-powered shopping agent does not charge a customer's payment method multiple times due to transaction retries when a payment gateway times out. The author explains how the problem arises from the non-idempotent nature of payment transactions, where retrying a failed transaction can result in double charges if not managed carefully. The solution involves using a durable function to wrap each transaction in a retry mechanism that memoizes successful attempts, preventing redundant charges by using idempotency keys provided by payment APIs. This approach allows the agent to handle transient errors without repeating successful transactions, ensuring a robust and customer-friendly payment process. The article emphasizes the importance of infrastructure design in preventing such issues and offers a practical guide for developers to implement these solutions using a mock payment tool and specific coding strategies.

In the evaluation of AI agents, offline and online methods serve distinct purposes and are often used in tandem to ensure robust performance monitoring. Offline evaluations involve testing the AI against a fixed dataset before deployment, functioning like unit tests to catch regressions and provide a controlled assessment environment. Conversely, online evaluations occur in real-time, assessing the AI's performance against live production data and user interactions, offering a dynamic and authentic measure of how the agent behaves in real-world scenarios. The combination of both methods allows teams to identify issues before release with offline evals, and to gain insights into the agent's real-time performance and user acceptance with online evals. The cost and complexity of these evaluations depend more on the scoring methods employed—such as algorithmic, signal-based, or LLM-as-judge—than on whether they are conducted online or offline.

In this detailed blog post, Linell Bonnette explores a complex issue encountered in TypeScript when using multiple middleware with the Inngest client, which causes the TypeScript types to break. The author discovers that when two middleware are applied, the return type of step.run unexpectedly collapses to an empty object due to the serialization and transformation process involving Jsonify, which composes multiple times when middleware is stacked. This issue arises from an optional property that introduces an undefined key in TypeScript’s mapped types, leading to an ill-formed type that the compiler silently accepts. Despite previous community attempts to resolve the problem, the author identifies that the root cause lies in where undefined enters the union of keys and proposes a fix by excluding undefined from the union, which effectively prevents type corruption. Further refinement is suggested by employing key remapping with 'as' to eliminate the need for additional exclusion steps, ensuring a more robust and simplified solution. This nuanced investigation highlights the subtleties of TypeScript's type system and the importance of testing type compositions thoroughly.