August 2024 Summaries

On August 16, 2024, Inngest experienced an outage that prevented function execution from successfully running due to the SDK Gateway's inability to handle requests properly. The issue was fixed on each instance to bring them back online and prevent future failures of this type. The root cause was traced to the SDK Gateway services log, which ran out of disk space due to increased load and infrequent logrotate running. The timeline for resolving the issue included manual intervention to clear disk space, rotating internal service discovery IPs, scaling down executor services, issuing new static IP addresses, and adding additional SDK Gateways for extra capacity and redundancy. The impact of this outage was a 46-minute function execution downtime and degraded performance on Vercel for 4 hours and 54 minutes. Corrective actions included improving alerting via Datadog, enhancing log rotation in the instance AMI, increasing SDK Gateway replicas, staggering service rotations, rehearsing with engineers to restore issues with SDK Gateways, and purchasing a /24 block of public IP addresses for future use.

Durable Functions extend Next.js capabilities by enabling long-running tasks without timeouts or managing queues and state. They power core features such as integrations, email scheduling, or AI workflows. Teams at Resend and Mintlify have built their backend workflow parts on top of Next.js using Durable Functions. These functions are composed of triggers and steps, which manage the state over time and overcome timeouts. Flow Control features like rate limiting, priority, or throttle configuration help control how runs span in time and avoid reaching the rate limit of third-party APIs.

Inngest is redesigning its UI based on community feedback to improve navigation and overall user experience. The new design focuses on enhancing the Information Architecture (IA) of the Cloud Application and Dev Server, addressing common pain points such as difficulty navigating and finding specific parts of the application. Through extensive user research and iterative testing, Inngest's Design team has developed a cohesive theme with clearer navigation, aligning the IA with users' mental models and providing high-level observability. The redesigned IA is scalable to support future functionalities and offers consistent navigation across both the Dev Server and Cloud Application.