August 2022 Summaries

At the Ray Summit, Anyscale unveiled new innovations in Ray 2.0 and its managed Ray compute platform to ease building and scaling AI applications. The event featured speakers from various industries discussing their use of Ray for advanced applications such as personalized content recommendations, game balancing, and optimizing FIFA World Cup Qatar 2022 logistics. Anyscale is a fully-managed, enterprise-ready unified compute platform that makes it easy to build, deploy, and manage scalable AI and Python applications using Ray, allowing organizations to bring AI products to market faster at significantly lower costs.

Ray 2.0 is a major release that aims to make distributed computing scalable, unified, and open. The new Ray AI Runtime (AIR) simplifies the process of running machine learning workloads by aligning existing native ML libraries and integrating with popular ML frameworks in the community. Additionally, Ray now supports natively shuffling large amounts of data with the Ray Datasets library. Production support for Kubernetes is provided through KubeRay, which makes it easier to deploy Ray-based jobs and services on K8s. High-Availability for large-scale Ray Serve deployments is also introduced in this release. New observability tooling provides developers with visibility into the health and performance of their Ray workloads. Deployment Graph API simplifies building, testing, and deploying an inference graph of deployments. RLlib refactors its algorithms to follow simpler patterns and introduces new algorithms for offline reinforcement learning. Overall, Ray 2.0 aims to make distributed computing more accessible and efficient for ML practitioners and infrastructure groups.

Anyscale, the company behind Ray, a unified framework for scalable computing, has announced Ray 2.0 and its enterprise-ready managed platform at the Ray Summit. The adoption of Ray is driven by the growing gap between the demands of machine learning applications and the limitations of single processors or servers. Thousands of organizations already rely on Ray for their AI initiatives. Anyscale also secured $99 million in Series C funding, co-led by existing investors Addition and Intel Capital. The company unveiled major new developments in Ray 2.0, including Ray AIR, KubeRay, integration with the ML ecosystem, ease of development & increased scalability, and its enterprise-ready Ray platform.

Ray is an open-source distributed compute framework developed at UC Berkeley's RISELab that enables users to run Python code in a parallel fashion across multiple machines. It provides a general-purpose clustering and parallelization framework that can be used to build and run any type of distributed application. Ray allows users to focus on building their ML use case, not distributed technologies. Arize is an ML observability platform that helps ML practitioners tackle issues such as model performance degradation, data drift, and data quality issues in real-time. Arize provides automated monitoring, strong troubleshooting workflows, and is built for scale, intuition, and ease of use. By combining Ray and Arize, users can offload tasks such as distributed computing and model monitoring to technology, freeing up time to focus on building high-value ML models using deep business domain knowledge.

This year's Ray Forward conference highlighted exciting stories of Ray's adoption in China, including hyper-scale applications at Ant Group, production deep learning on Ray at DJI, next generation Ray-based ML Platform at Qihoo, and more. The Ray Community in China has been organizing a "Ray Forward" conference since 2020 as a companion event to Ray Summit. Key observations include the growing number of vertical computing systems, which are becoming increasingly complex, and Ray emerging as an ideal abstraction layer to simplify these complexities. Ant Group used Ray to power their online serving and new use cases such as audio/video processing, while DJI utilized Ray for distributed deep learning in production due to its flexibility and low overhead. Huawei Cloud launched a new framework called Fathom (powered by Ray), which provides seamless development experience on large scale distributed Data + AI problems. Qihoo 360 introduced an ML toolkit Veloce (built on Ray), providing flexible heterogeneous hardware support, rapid development iterations, and stronger cross-stack optimizations than their existing Kubeflow-based system. Byzer integrated SQL data processing, model training, and serving in a flexible, efficient, and scalable way using both Spark and Ray. The conference also showcased the adoption of Ray by companies such as Baihai Tech and Byzer, highlighting its ability to simplify development to deployment flows and provide optimized development pipelines.

In this blog post, we explored the options for deploying machine learning (ML) models in production. Python has become a popular language for data science, but generic Python web servers such as FastAPI are also gaining traction. FastAPI is a modern, fast web framework that aims to optimize the developer experience using an integrated development environment and leveraging common standards. It offers features like high performance, faster coding speed, fewer bugs, intuitive editor support, easy use, robust production-ready code, and standards-based APIs. In contrast, generic Python web servers were designed for microservices but not specifically for ML models. Specialized ML serving libraries have emerged to optimize throughput without sacrificing latency, featuring techniques like model compilation, microbatching, bin packing, and "scale to zero" autoscaling. Ray Serve provides a performant Python web server and a specialized ML serving library, allowing developers to plug in a web server such as FastAPI easily and take advantage of its features combined with Ray Serve's own features, including the ability to compose multi-model inference pipelines, scale independently on different hardware, and configure the number of replicas.