Optimizing QStash For Millions of Schedules
Blog post from Upstash
Upstash Workflow and QStash redesigned their scheduling architecture after growth from roughly 50,000 to more than 700,000 schedules exposed the limits of storing active cron state in process memory. The prior system loaded schedules from Redis and rebuilt in-memory cron entries on each process restart, causing restart time to scale with schedule volume and reach 17 minutes in one deployment; benchmarking also revealed inefficient next-run lookups in the cron library. Although switching libraries improved performance, it could not address the underlying scalability issue. The new approach reuses QStash’s existing durable Redis-backed delayed-message pipeline by representing each schedule as a task that, when due, publishes its scheduled message, calculates its next execution time, and enqueues itself again. This self-rescheduling design eliminates the need to preload schedules during startup, makes restarts independent of total schedule count, and uses worker partitioning and durable external state to support more predictable operations at millions of schedules.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.