I've operated petabyte-scale ClickHouse ® clusters for 5 years

The author shares extensive insights and lessons learned from years of managing ClickHouse clusters, focusing on the challenges and best practices for maintaining such systems, particularly in the context of a company like Tinybird. They highlight the ease of setting up a ClickHouse cluster but emphasize the complexities of keeping it operational, especially when dealing with petabyte-scale clusters and high query loads. The discussion covers the architectural choices, such as the use of replicas and shards, and the shift towards cloud storage for cost efficiency and management advantages. The author also delves into the intricacies of data ingestion, pointing out common pitfalls like data duplication and system overloads, and suggests strategies for balancing batch sizes and managing merges. Additionally, the text touches on the costs associated with running ClickHouse clusters, the need for skilled personnel to manage these systems, the importance of careful configuration, and the challenges of upgrading the database without downtime or data loss. There is also a focus on the limitations of ClickHouse's cloud storage capabilities compared to other systems, and the importance of testing configurations and monitoring system performance to ensure stability and efficiency.