March 2026 Summaries

Background job processing often starts with simple queuing systems like BullMQ, Celery, or Sidekiq, which are well-suited for atomic, self-contained tasks such as sending emails or processing files. However, as job complexity grows — requiring multi-step processes, indefinite waits for external signals, or handling retries without redoing costly operations — durable execution platforms like Temporal, Inngest, or AWS Step Functions become more appropriate. These platforms guarantee task completion by persisting execution state and enabling jobs to resume from the point of failure, offering inherent observability and flow control. For example, in a complex AI lead processing pipeline involving multiple APIs and human approval, a durable execution platform efficiently manages retries, state, and long-running tasks without the need for additional infrastructure to track progress or coordinate steps. While the rise of AI necessitates more durable solutions due to longer, stateful, and human-in-the-loop processes, simpler, atomic tasks can still be effectively managed by traditional queues if they can restart safely from the beginning.

Building a production AI image app involves handling two main challenges: inference and workflow orchestration. The inference aspect, efficiently managed by fal.ai, requires running models quickly and reliably at scale, offering access to numerous models through a single API while managing GPU provisioning and scaling. Workflow orchestration, addressed by Inngest, involves coordinating the processes around the inference call, ensuring tasks like resuming execution after inference and retrying failed uploads are handled without redundant computations. The integration of these tools facilitates the creation of a seamless media production pipeline, where fal.ai manages asynchronous image generation via a queue API, and Inngest coordinates the execution steps with full observability and minimal compute costs. This setup allows for efficient image generation and storage, protects inference costs through step-level retries, and maintains per-user fairness by dynamically managing queues, ensuring that large batch jobs don't hinder real-time requests. As the product grows in complexity, additional steps such as post-processing or custom model training can be added without altering the underlying architecture, highlighting the adaptability and scalability of the system.

Inngest transformed its cron scheduling system from a fragile, database-dependent polling model into a robust, self-healing, and horizontally scalable distributed scheduler using its custom-built queueing infrastructure. Initially, Inngest's cron service relied on polling the database for scheduled functions, but as usage increased, this approach became unsustainable due to database slowdowns and memory crashes causing missed executions. By leveraging its existing queue system, Inngest restructured cron scheduling to maintain a single pending job per scheduled function in the system queue, which schedules subsequent jobs upon execution. This approach ensures durability, as jobs persist beyond process crashes, and reliability, with built-in support for delayed execution and delivery guarantees. The migration process involved a seamless transition where both the old and new systems ran concurrently, facilitated by feature flags and testing with pilot accounts to ensure accuracy and performance. The new architecture has improved observability and reduced support demands by integrating cron scheduling into the established queue system, offering a unified, dependable framework for all time-based orchestration tasks.

Node.js operates on a single-threaded model, which is generally efficient for handling I/O concurrency through its event loop, but can lead to problems when CPU-intensive tasks monopolize the main thread, causing event loop starvation. This issue was encountered with Inngest Connect, a persistent WebSocket connection that experienced "no available worker" errors due to CPU-heavy user code blocking heartbeats. The solution involved shifting Connect internals to a worker thread, enabling isolation of critical path operations from user workloads. Worker threads in Node.js, unlike concurrency primitives in other languages, require communication through message passing, as they can't directly execute functions passed to them. Each worker, being a separate V8 isolate, communicates with the main thread via serialized messages, which, while adding overhead, ensures independent event loops. This approach resolved the starvation problem but required careful handling of communication protocols and management of worker thread lifecycle, including respawning with exponential backoff to handle potential worker crashes. Despite these challenges, the use of worker threads effectively eliminated the errors and maintained robust WebSocket connections, demonstrating their value in scenarios where main thread yield cannot be guaranteed.

Windmill, an HR and performance platform, developed a personal RevOps agent named Pim to replace their unreliable n8n-centric workflow, aiming to integrate various tools like Attio, Slack, and PostHog into a cohesive system. The project was inspired by OpenClaw but tailored for company-wide use, prioritizing security and reliability through a set of foundational rules, such as no open internet ports and creating context files for clear tool usage. Utilizing Inngest Connect, which leverages persistent outbound WebSocket connections, Windmill avoided exposing their network while allowing Pim to handle tasks like lead processing, sales call summarization, and internal communication through Slack. The creation of context files before tool implementation enabled Pim to have a comprehensive understanding of the company's operations, ensuring efficient and accurate task execution. This approach resulted in improved operational workflows and trust within the team, with Pim now being a dependable element of their business processes.

Building an AI agent using Inngest offers a robust solution to overcoming common production challenges such as tool call failures, rate limits from LLM providers, and task timeouts. The process involves creating a loop that enables the AI agent to think, act, and observe by using three key primitives: step.run() for durable task execution, step.invoke() for synchronous delegation to sub-agents, and step.sendEvent() for asynchronous task delegation. These primitives ensure that each step in the agent's workflow is tracked, retryable, and resumable, allowing the function to continue from the last completed step after a crash, thus maintaining durability. The agent loop, which is a returned function for reusability across different functions, handles tasks such as loading context, managing user interactions, and executing tools. Inngest's dashboard provides detailed observability of each function run, offering insights into step sequences, inputs, outputs, and performance metrics. This setup not only facilitates debugging and error handling but also allows for the delegation of long-running tasks to sub-agents, enhancing the flexibility and scalability of AI-driven workflows.

The author reflects on their journey with TypeScript, initially embracing its benefits in making code cleaner and clearer compared to JavaScript, but realizing their limitations during a major refactor of the Inngest TypeScript SDK. They discovered that creating types for a library differs significantly from developing applications, as public types in an SDK serve as a contract with users, requiring careful consideration of user experience, documentation, and potential breaking changes. The author emphasizes the importance of designing types that are clear, intuitive, and maintainable, highlighting the necessity of empathy in anticipating how users will interact with the SDK. Through this process, they encountered new challenges, such as transforming user-facing types to align with internal logic without exposing complexities, and the broader implications of Hyrum's Law on type systems. Ultimately, the experience underscored the need for a shift in mindset from merely ensuring code compiles to ensuring it is understandable and usable for others, a valuable perspective in any software development context.

In agentic systems, three delegation patterns are essential: synchronous (sync), asynchronous (async), and scheduled delegation, each serving specific purposes and offering distinct advantages for managing tasks. Sub-agents, which are separate execution contexts spawned by a parent agent, facilitate context compression, allowing the parent agent to remain efficient across long tasks and multiple requests. Sync delegation is needed when the parent requires an immediate result to proceed, while async is ideal for independent tasks that don't require immediate feedback, and scheduled delegation is used for tasks that need to be executed at a specific future time. The use of sub-agents significantly reduces the token burden on the parent by summarizing lengthy or complex tasks, thereby enhancing the system's overall efficiency and responsiveness. The text emphasizes the benefits of maintaining a generic approach to agentic systems rather than diving into specialization, which can lead to unnecessary complexity, and highlights the importance of flexibility and simplicity in adapting to evolving models and patterns.

Inngest implemented a new state management system to address the limitations of its previous Redis cluster, which was overwhelmed by read and write operations. The solution involved introducing a state proxy service that uses a caching layer to alleviate load on the Redis system by reducing redundant reads, particularly for immutable data like triggering events and step inputs/outputs. This proxy service, deployed as a Kubernetes headless service, employs consistent hashing for efficient routing and uses PebbleDB for caching due to its lower memory allocation requirements compared to Badger. The rollout was conducted in stages, with a shadow mode and per-account feature flags to ensure stability and allow for quick rollbacks if necessary. As a result, Redis read operations decreased by 67%, and the network overhead was significantly reduced, while improved observability and control over I/O timeouts were achieved. The abstraction provided by the proxy also facilitates future migration to a more scalable database system without requiring changes to other services.

In Next.js AI pipelines, route handlers often fail due to their lack of memory, causing the entire process to restart from zero upon any failure, which can be costly and inefficient at scale. Traditional solutions, like increasing execution limits or enabling retries, often extend the problem rather than resolve it. The key to reliable multi-step AI pipelines is implementing step-level durability, allowing each step to be independently durable and preventing the need to re-run successful steps upon failure. Inngest Durable Endpoints offer a solution by enabling this durability without altering existing route structures, thus ensuring that only failed steps are re-executed, reducing duplicated inference costs, and maintaining efficiency. This method provides a more reliable execution model crucial for scalable agent orchestration, distinguishing between real-time interactive agents and background processing tasks. As agent architectures become more complex, these durable endpoints provide a foundational improvement to handle multi-step processes effectively, although certain features like POST body support and flow control are still under development.

Managing concurrent tasks around shared states in Python's asyncio presents challenges that standard library tools like asyncio.Event and asyncio.Condition do not fully address under real concurrency pressure. In the context of developing Inngest's Python SDK, the challenge was to manage multiple asynchronous handlers coordinating around WebSocket connection states, requiring a mechanism to drain pending requests during specific state transitions. Initial attempts using polling, asyncio.Event, and asyncio.Condition revealed limitations such as inefficient CPU usage, the need for multiple event objects, and issues with fast state transitions leading to missed updates when using conditions. The proposed solution involves using per-consumer queues, where each transition is buffered into a queue, ensuring no state transitions are missed. This approach, implemented via the ValueWatcher class, offers a thread-safe method for observing value changes with asynchronous watchers, addressing issues of thread safety, atomic registration, and resilient notifications. The solution is particularly effective for managing shared mutable state in asyncio applications, providing a robust method to coordinate tasks and states without missing critical transitions.

The Inngest TypeScript SDK v4 introduces a comprehensive overhaul with key enhancements including a rewritten middleware system, type-safe trigger helpers, structured logging, and improved parallel step execution. The middleware system now utilizes a class-based approach with lifecycle hooks for better integration across function execution phases, while type-safe trigger helpers replace the centralized EventSchemas, allowing event types to be defined alongside functions for scalability and runtime validation. Structured logging adopts an object-first format, supporting custom loggers, and the optimizeParallelism feature is enabled by default for more efficient parallel workloads. Additionally, checkpointing is automatically activated, reducing latency in sequential tasks, and the API has been streamlined for greater consistency, with changes such as lazy initialization for edge environments and modifications to function triggers and server options. Upgrading to v4 involves notable breaking changes, but the transition is guided by a detailed migration guide to assist developers in adapting to the new architecture.

Inngest has enhanced its Traces feature to improve the debugging of asynchronous workflows by providing detailed execution data directly on the frontend. The updates include timing breakdowns that identify queue delays, Inngest processing time, and flow control waits, allowing users to pinpoint inefficiencies without guesswork. Extended traces now incorporate OTel spans from user instrumentation, enabling a comprehensive view of database queries and API calls within the execution timeline. Additionally, AI step metadata offers insights into model, token counts, and latency, helping to detect issues like prompt bloat early. These improvements, which began rolling out on February 24, aim to deliver a clearer picture of workflow performance and facilitate easier debugging by providing detailed error contexts and execution insights.

Utah, the Universally Triggered Agent Harness, is an event-driven infrastructure designed to connect, protect, and orchestrate components for agent runtimes without executing tasks itself. It leverages Inngest Cloud to handle orchestration, providing observability, reliability, and multi-player distributed agent orchestration. The architecture decouples orchestration from the agent loop, allowing for flexible triggers and durable execution, while maintaining context through a think → act → observe loop, where each step is an independently retryable unit of work. This harness supports various tools and sub-agent delegation, using existing libraries for tasks such as file operations and shell execution, and is designed to handle issues like context management and multi-provider LLM support. Utah also introduces singleton concurrency to manage conversation contexts and explores future enhancements for multi-player functionality and real-time updates. The project emphasizes infrastructure solutions for AI agent challenges and encourages community collaboration through its open-source repository on GitHub.