March 2021 Summaries

Ant Group has implemented a flexible, high-performance, stable, and scalable online resource allocation system based on Ray to support the largest online shopping event in the world, Double 11. The system's deployment scale reached more than 6000 CPU cores and is currently used for various application scenarios including marketing and order allocation. The online resource allocation solution involves a complex engineering implementation relying on offline and real-time data, with the algorithm's implementation relying on both real-time and iterative calculations. Ray provides a simple and easy-to-use API, supports convenient resource scheduling, and has second level fault-tolerant recovery ability, ensuring the availability of the service. The online resource allocation solution based on Ray has been running stably in Ant Group, successfully supporting large-scale activities such as Double 11 and Double 12.

A distributed shuffle is a data-intensive operation that requires a system built specifically for the purpose, but can be expressed in just a few lines of Python using Ray, a general-purpose framework whose core API contains no shuffle operations. Shuffling a small dataset is simple enough to do in memory, but larger datasets bring new challenges such as multiple nodes, spilling to external storage, and enforcing memory limits. A common method for shuffling a large dataset is to split the execution into a map and reduce phase, with the data being shuffled between the two phases. Ray manages parallelism and task dependencies for small shuffles, but for larger shuffles, it requires scheduling in waves due to limited memory capacity, spilling objects to external storage, and restoring them during the reduce phase. Ray's scheduler takes into account both CPU and memory availability to prevent deadlocks and enforce memory limits. The framework provides a high-performance shared-memory object store, spills objects transparently to external storage, applies admission control during task scheduling, and is designed to scale up to large datasets such as sorting 100TB.

The pandas library provides easy-to-use data structures and tools for data analysis but can be slow with large amounts of data. Modin is a library that mitigates these issues by providing benefits such as scalability, integration with the Python ecosystem, and Ray clusters. Modin's coverage of the pandas API is over 90% and focuses on commonly used methods like pd.read_csv, pd.DataFrame, df.fillna, and df.groupby. To get started with Modin, one needs to install it and change the import statement from `import pandas as pd` to `import modin.pandas as pd`. Modin can speed up pandas workflows by utilizing multicore/multinode support, system architecture, and ease of use. Modin's API covers most of the pandas API, but if a method is not implemented in Modin, it defaults to a pandas implementation. Modin has an optimal system architecture that includes APIs layer, Query Compiler layer, DataFrame layer, and Execution layer, with Ray being the default execution engine. Modin allows users to scale their code to multiple nodes, making it easier to work with large datasets.

Ray is an open source library for parallel and distributed Python that can be paired with PyTorch to rapidly scale machine learning applications, offering simplicity, robustness, and performance advantages. It has a general-purpose framework that has built many libraries and frameworks on top of it to accomplish different tasks, with the vast majority supporting PyTorch, requiring minimal modifications to code, and integrating seamlessly with each other. RaySGD is a library that provides distributed training wrappers for data parallel training, offering ease of use, scalability, accelerated training, fault tolerance, and compatibility with other libraries like Ray Tune and Ray Serve. Ray Tune is a Python library for experiment execution and hyperparameter tuning at any scale, allowing launch of multi-node distributed hyperparameter sweeps in fewer than 10 lines of code, with first-class support for GPUs and automatic management of checkpoints and logging to TensorBoard. Ray Serve is a library for easy-to-use scalable model serving, offering compatibility with many other libraries like Ray Tune and FastAPI. RLlib is a library for reinforcement learning that offers both high scalability and a unified API for a variety of applications, including native support for PyTorch, TensorFlow Eager, and TensorFlow, as well as support for complex model types and multi-agent algorithms. The Ray Cluster Launcher allows users to launch and scale machines across any cluster or cloud provider with ease, automating tasks such as autoscaling, syncing files, submitting scripts, port forwarding, and more.