Under the hood, off the hot path: how D1 scales to millions of databases

Cloudflare describes how it redesigned the control plane for D1, its managed serverless SQLite database, to support millions of small databases created for users, tenants, preview environments, and AI-assisted development platforms, amid 15-fold annual growth in live databases and 18-fold growth in daily active ones. Each D1 database is backed by a SQLite Durable Object, while edge Workers handle APIs and queries, Workers KV stores globally distributed routing metadata, Hyperdrive provides cached access to central Postgres metadata, and ClickHouse supports analytical and billing workloads. The system separates latency-sensitive query execution from centralized control-plane functions so most queries remain at the edge and avoid Postgres, with Worker bindings able to route directly to the query proxy and Durable Object. Database creation initializes a Durable Object near the user where possible, records its authoritative mapping in Postgres, and populates KV, while retries, periodic heartbeats, and idempotent workflows repair failed or incomplete operations and remove orphaned objects. For usage limits, D1 asynchronously collects storage updates through queues, pre-aggregates account totals to prevent expensive scans and write contention, and uses cached account-level reads for enforcement; separate edge-distributed limit checks enforce daily free-tier query quotas. The design emphasizes failure recovery, eventual consistency where strict synchronization is unnecessary, realistic load testing, and selecting different storage systems for transactional metadata, globally replicated routing, and large-scale analytics.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.