Supercharging your large ClickHouse data loads - Tuning a large data load for speed

To achieve maximum ingestion performance with ClickHouse, it's essential to choose the right gear (insert block size) and acceleration level (number of parallel insert threads) based on the available horsepower (CPU cores and RAM). The ideal combination depends on trade-offs between these factors, including resource contention and background merge threads. By experimenting with different settings and analyzing performance metrics, it's possible to find a sweet spot that maximizes ingestion speed. A formula can be used to calculate the optimal settings for top speed, which involves choosing half of available CPU cores for insert threads and an intended peak memory usage. Additionally, ClickHouse Cloud's SharedMergeTree table engine and seamless cluster scaling enable linear scalability, allowing large data inserts to be run as fast as required by adding more servers or CPU cores.