September 2023 Summaries

Dolphin AI Assistant provides an overview of durable workflow engines, which allow developers to write reliable code without worrying about queues or state management. These engines are designed to be code-first and should get out of the way, letting developers focus on their code. Workflow engines provide primitives such as steps (single transactions that must complete as close to exactly once as possible) and sleeps (the ability to wait a specific amount of time). They abstract some sort of queue under the hood and typically all do something similar: run workflows, check for new steps, update state, and re-run the function. Event-driven workflow engines combine semantics that you'd find in an event stream, allowing functions to automatically run any time specific events are received. These modern replacements for queues offer tooling and semantics above and beyond what raw infrastructure provides, improving how we build products day to day.

On August 31, 2023, a backlog of steps processing occurred due to event stream acknowledgement issues within Inngest's HTTP execution drivers. The issue was caused by invalid waitForEvent matching expressions, which led to an error and subsequent nack (not acknowledge) of certain messages. This caused extra load on several services, including the step processing handlers, pause/resume service, cancellation matching, and executor. Inngest introduced mitigations to fail functions when receiving invalid waitForEvent expressions, and over the next 36 hours, they deployed code to improve efficiency in various areas. They apologize for any delays or issues caused during this time period and have taken remedial actions to prevent such occurrences in the future.