January 2023 Summaries

Anyscale and Weights & Biases integration simplifies MLOps by addressing challenges such as data management, model performance, reproducibility, scalability, deployment, and security, providing a unified platform for managing and deploying machine learning models at scale with built-in consistency, reproducibility, and flexibility across the ML lifecycle. This integration allows for seamless navigation between Anyscale and Weights & Biases tools, enabling easy tracking of experiments, logs, and metadata, as well as toggling back to Anyscale for debugging and consistency, reducing friction throughout the ML development lifecycle.

We have achieved a new world record on the CloudSort benchmark using Ray, a general-purpose distributed execution framework, by developing Exoshuffle, a simple and flexible architecture for building distributed shuffle that achieves high performance. This record is 33% more cost-efficient than the previous one, set in 2016, and demonstrates Ray's ability to simplify high-performance distributed programming and achieve scalability and performance required for running challenging distributed data processing jobs. The success of Exoshuffle-CloudSort can be attributed to innovations in both open-source software and cloud infrastructure, including disaggregated storage that allows scaling compute and storage independently and elastically, and Ray's distributed futures abstraction that enables building complex distributed applications with fine-grained control. This achievement opens up the possibility for running a rich set of data processing workloads on Ray, and we are working to bring Exoshuffle capabilities into Ray Datasets to support use cases such as shuffling data loading for ML training.

We’re excited to share recent developments for Ray in the 2.2 release -- enhanced observability, improved performance for data-intensive AI applications, increased stability, and better UX for RLlib. The Ray community has driven requirements for performance, robustness, and stability, which have been addressed with this new release. Ray Jobs API is now GA, allowing users to submit locally developed applications to a remote Ray Cluster for execution. Improved observability features include the ability to visualize CPU flame graphs of worker processes and additional metrics in the Ray Dashboard. Additionally, RLlib's Algorithms feature flexible fault tolerance for simulation/rollout workers and evaluation workers, and improvements have been made to the Ray Tune library for better checkpoint syncing and retries. The release also focuses on reducing latency and memory footprint for batch prediction, with nearly 50% improved throughput performance and 100x reduced GPU memory footprint. Enhanced support for the ML data ecosystem has also been added, including expanded version compatibility with Apache Arrow and full TensorFlow TF records reading/writing support for Ray Data. Out-of-memory errors have been addressed by enabling the Ray Out-Of-Memory (OOM) Monitor by default, and dynamic block splitting is now enabled by default to address performance issues with large files. Finally, UX improvements have been made to RLlib's command line interface and checkpoint format, making it more cohesive and transparent.

The text discusses batch processing in data engineering and machine learning, specifically focusing on scaling model training with Ray Core APIs. It explains two approaches to conducting batch training: distributed data loading and centralized data loading. The first approach involves reading each independent task into memory to ensure the desired data fits within memory, while the second approach preloads data partitions into the Ray object store and extracts batches from it. The text also describes an optimized approach using Ray's central object store, which reduces training times by 3-5X compared to the previous approaches. It highlights the benefits of this optimization technique, including lower execution and training times, but notes that it may require more memory and CPU resources. Ultimately, the choice between these approaches depends on the specific use case and size of the dataset.

The Ray Summit, an annual conference for the Ray community, will take place in San Francisco, CA, on September 18-19 and 20th, featuring a two-day event with technical deep dives, use cases, lightning talks, and training. Proposals for conference talks are currently being accepted until February 17th, offering benefits such as sharing expertise, building public speaking skills, and networking opportunities. The conference focuses on topics like scalable ML workloads, reinforcement learning, cloud computing, and integrations with common ML libraries and frameworks. To submit a proposal, follow simple tips to ensure the value of your idea shines through, including focusing on a specific angle or technique, editing for clarity, and keeping the audience in mind.