June 2021 Summaries

Ray is a general-purpose distributed system designed to integrate data processing libraries into distributed applications, with a robust distributed memory manager. Dask-on-Ray is a community-contributed plugin that enables running Dask task graphs on Ray clusters. The blog post compares the memory management and performance of Ray versus Dask with its built-in scheduler, highlighting recent features in Ray such as object spilling and better control of memory usage. The comparison focuses on architectural differences between Dask and Dask-on-Ray, including the location of the object store, configuration parameters, and system reliability under memory pressure. Microbenchmarks demonstrate that Dask-on-Ray can scale sorting to 10x larger datasets than with Dask's built-in scheduler, while Ray's shared-memory object store provides a more precise measurement of total memory usage on each node. The design differences between Dask and Dask-on-Ray lead to varying performance characteristics, particularly when dealing with large datasets and memory-intensive workloads.

XGBoost-Ray` is a novel backend for distributed XGBoost training that leverages Ray to scale XGBoost training from single machines to clusters with hundreds of nodes while minimizing code changes. It enables multi-node and multi-GPU training, supports advanced fault tolerance handling mechanisms, and integrates seamlessly with the hyperparameter optimization library `Ray Tune`. The distributed training setup can be achieved by changing only three lines of code from a single node non-distributed XGBoost setup. The backend also provides a seamless integration with the Scikit-learn API, allowing users to act as a drop-in replacement for models such as `XGBRegressor` or `XGBClassifier`. XGBoost-Ray achieves similar performance as XGBoost-Spark and XGBoost-Dask in terms of speed, scalability, and peak memory utilization. It also offers full GPU support and advanced fault tolerance handling mechanisms. The backend is developed in collaboration with Uber and is available for more information on the GitHub repository.

Ray is a distributed computing framework that integrates with various libraries and frameworks, providing benefits such as fine-grained control over worker scheduling and fault tolerance, the ability to run subroutines within larger programs, and passing data in-memory to downstream stages. Ray's Level 1 integration allows for scheduling only, with communication done outside of Ray, which is suitable for libraries that have mature internal communication stacks but want additional features. This level of integration provides benefits such as low latency communication, parallelization of computation into many individual tasks, and coordination of complex actor or task topologies. Libraries may also leverage Ray's distributed object store, which offers features like first-class object references, shared-memory support, and object spilling. The level of effort to integrate varies depending on the library, with Level 1 integration typically starting at fewer than 200 lines of code, and Level 2-3 integration ranging from straightforward to high effort. Ray libraries can work together using patterns such as running one distributed library as a subroutine of another, passing data between libraries, or simply invoking multiple libraries in separate tasks and actors.