Introducing multi-stage distributed query execution in ClickHouse Cloud

ClickHouse Cloud introduces multi-stage distributed execution to enhance query scalability and performance, addressing bottlenecks in large joins and high-cardinality aggregations by repartitioning intermediate data between execution stages. This new approach allows queries to be executed in parallel across multiple nodes, removing limitations of previous models that restricted scalability for modern petabyte-scale workloads. The method employs exchange operators to efficiently move data between stages, optimizing query execution by allowing for dynamic repartitioning and parallel processing. TPC-H benchmark tests demonstrate significant speed improvements, with some queries achieving up to 7.4× faster execution on eight nodes compared to a single node, although certain single-node optimizations are not yet fully supported. The development of a cost-based optimizer is anticipated to further enhance performance by automatically selecting the best aggregation strategies based on various factors. Currently, multi-stage distributed execution is available in an experimental phase within ClickHouse Cloud's private preview program.