Running out of runway: migrating Temporal Cloud's billing store to ClickHouse

Temporal migrated its metering and billing read paths from a previous warehouse to ClickHouse to address limited growth capacity, costly scheduled deduplication, and delayed customer usage data, while leaving ingestion and streaming infrastructure unchanged. Because the at-least-once delivery pipeline intentionally creates duplicate events and billing inaccuracies can directly affect invoices, the team shadowed both databases for six weeks, serving customers from the legacy system while comparing production-scale results in the background. They defined tolerated floating-point variation, distinguished expected freshness differences caused by ClickHouse’s read-time deduplication, and treated implausible results as blocking defects; this process exposed a 64-bit integer overflow that produced negative usage totals for large accounts. The rollout proceeded in reverse data-flow order, moving reversible customer-facing read APIs before the harder-to-reverse billing aggregation path, with per-account flags enabling rollback. The migration increased projected capacity headroom from about one year to three to five years, reduced steady-state costs by roughly 70%, improved query performance, eliminated a batch dependency and its operational burden, and made usage data available to customers around 30 minutes sooner.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.