Shuffle V2 in Ray Data: Faster, Fault-Tolerant Joins and Aggregations

Ray Data’s Shuffle V2 redesigns how key-based operations such as joins, groupbys, repartitions, and deduplication handle intermediate data, replacing V1’s long-lived aggregator actors with separate map and reduce stages connected through materialized shards in the Ray object store. This approach allows intermediates to spill to disk, use Ray’s lineage-based reconstruction after failures, measure actual partition sizes to reduce out-of-memory risks, and release compute resources between stages rather than reserving them for an entire shuffle. V2 also introduces input coalescing to reduce object counts, configurable Arrow IPC compression, downstream map fusion, and Arrow-native vectorized aggregations for several common functions. Benchmarks on TPC-H SF1000 workloads report major gains for aggregation-heavy queries and successful completion of joins that V1 could not finish, although V2’s architecture alone can be slower for small shuffles due to encoding and object-store overhead. Available experimentally in Ray 2.58 as `HASH_SHUFFLE_V2` and renamed `SHUFFLE_V2` in Ray 2.59, the engine is intended to eventually support sort and random shuffle workloads, while planned disk shuffle and incremental join features aim to further reduce memory and object-management limits.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.