Scaling ClickHouse ® ingestion with multi-writer architecture

Tinybird's platform faced a significant scaling challenge when one of their largest customers' data volumes began exceeding the capacity of a single ClickHouse writer instance, which necessitated the development of a multi-writer ingestion mode to distribute the load across multiple instances. This new mode, designed for real-time data ingestion, allows specific tables or workspaces to be routed to different ClickHouse instances within the same cluster, aiming to alleviate pressure on individual writer instances and ensure cluster stability. By opting for a static routing approach with Varnish load balancer and a custom Varnish extension, Tinybird achieved a system that prioritizes simplicity, reliability, and predictability, enabling them to manage high-throughput data ingestion more effectively and scale with customer demands. The implementation has been rigorously tested and gradually rolled out to larger customers, successfully managing peak data loads and demonstrating the platform's capacity for future growth.