How we extended Apache DataFusion to execute one query across many machines

Datadog created Distributed DataFusion, an open-source Rust framework that extends Apache DataFusion from single-machine execution to low-latency distributed query processing across multiple machines. The project supports Datadog’s move toward a unified, composable data stack based on Apache Arrow, Substrait, and DataFusion, reducing reliance on separate specialized engines for metrics, logs, traces, and other observability data. Unlike existing distributed systems such as Spark, Trino, ClickHouse, and Ballista, it was designed for extensibility, Arrow and Substrait integration, zero-copy streaming, and efficient interactive workloads. It distributes DataFusion physical plans into stages and tasks, using mechanisms such as NetworkShuffle to repartition data between workers for operations like grouped aggregations and NetworkCoalesce to collect distributed results. The framework can retain lightweight queries on one machine to avoid network, serialization, and coordination costs, while enabling heavy queries to run across workers; Datadog reports some production queries complete up to 10 times faster at the same overall cost. Distributed DataFusion is available under the Apache 2.0 license and is being further developed with adaptive query execution and potential GPU acceleration, while allowing users to supply their own data sources, networking, execution nodes, and planning logic.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.