July 2021 Summaries

Ray Summit 2021 featured a variety of technical sessions showcasing the capabilities of Ray, including machine learning, reinforcement learning, data processing, and scalable Python use cases. Industry experts from top companies such as Uber, Shopify, and Robinhood discussed their experiences with Ray in production ML platforms, highlighting its benefits for distributed training, hyperparameter optimization, and fault tolerance. The event also featured talks on the development of open-source AutoML frameworks like Ludwig, which can be scaled to train models on massive datasets across hundreds of machines in parallel using Ray. Additionally, Anyscale was showcased as a fully managed alternative to Ray's Cluster Launcher, enabling seamless application lifecycle management from dev to prod.

The Ray Summit 2021 featured numerous technical sessions on various topics, including machine learning, deep learning, reinforcement learning, and data processing. Some of the most popular talks highlighted the use of Ray for scalable Python and its ecosystem. Notably, the ML Platform on Ray panel discussed the state of machine learning within organizations like Uber, Shopify, and Robinhood, while "Distributed XGBoost on Ray" showcased how companies are using XGBoost-Ray within their internal machine learning infrastructure. Other talks focused on scalable ecosystem restoration with Dendra Systems' tools and AutoML framework Ludwig on Ray. The event also highlighted the use of Anyscale as a fully managed alternative for scaling Ray applications.

Ray can be paired with Apache Kafka to power streaming applications. Data processing needs are far outpacing the development of faster hardware, driving a transition from vertical scaling to horizontal scaling. Event streaming platforms have emerged to capture and react to events in real-time, often acting as distributed logs that allow applications to produce and consume data horizontally. However, distributing stream processing to keep up with production rates is not easy, as the rate of production can vary drastically, requiring manual sizing that often results in suboptimal behavior due to wasted resources or slow reactions to peak demands. Ray Clusters can alleviate these issues by autoscaling to meet the demands of a stream processing job, ensuring efficient use of resources and delivering a "serverless" experience.