Attack of the Shards: Our Migration to FoundationDB, Part 2

Inngest’s second update on its migration from Valkey to FoundationDB describes a redesigned queue architecture intended to improve scalability, account fairness, and operational flexibility. The system replaces a three-level account, function, and tenant hierarchy with a two-level structure of function pointers and optional tenant queues, while logical shards group roughly 20,000 accounts to keep FoundationDB index scans manageable. A metadata service and Redis cache route newly enqueued work to each account’s assigned shard, and account migrations either update ownership metadata within a shared database or copy items safely between databases using stable IDs and duplicate handling. Multiple FoundationDB executors can concurrently discover and lease work through transactions, while weighted and sequential scanning strategies seek to prevent high-volume accounts from dominating execution and to reduce starvation. Although the new design requires careful rollout and tuning, it shifts large backlogs from costly memory to disk and allows work to move between executor pools without copying queued data when shards share a database.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.