December 2022 Summaries

Foobot is an AI-based solution that optimizes building energy efficiency by training fully autonomous control agents for Heating, Ventilation and Air Conditioning (HVAC) systems, reducing energy consumption and greenhouse gas emissions. The solution is versatile, scalable, and proven to bring large energy savings in small to large office buildings, hospitals, or schools. Foobot uses software and AI to address the challenges of energy cost crisis and environmental reduction by creating digital twins of buildings and using reinforcement learning (RL) agents to optimize HVAC controls. The company's training platform was significantly improved with the use of Ray Tune and RLlib, which enabled distributed parallel training across multiple CPUs and machines, reducing training time by 70% and accelerating time-to-market by 2x.

Ray is a high-performance distributed computing framework that enables companies to scale machine learning training workloads by up to 10x compared to existing tools like Celery, AWS Batch, SageMaker, Vertex AI, Dask, and more. Ray's flexible scheduling and unification capabilities make it an ideal solution for training many models, as demonstrated by companies like Instacart, Ecommerce, and B2B analytics firms that have seen order-of-magnitude performance and scalability wins using the framework. By leveraging Ray's built-in libraries and resource-based scheduling, developers can efficiently train multiple models in parallel, reducing training times and improving overall performance. Additionally, Ray integrates with other machine learning ecosystems and frameworks, including PyTorch, TensorFlow, Horovod, XGBoost, Scikit-learn, Hugging Face, and LightGBM, making it a versatile solution for building scalable machine learning workflows.

Ray is a unified framework that makes building machine learning platforms easier than ever by providing scalable libraries and tools for Python developers to build and scale their applications. It has an ecosystem of standalone libraries that can be used to address some of the key features required in modern AI applications, including bridging the gap between development and production, easy ML scaling, and ecosystem integration. Ray is particularly useful for teams struggling with managing the machine learning lifecycle, as it provides a suite of scalable machine learning libraries that can be used and composed like any other Python library, simplifying the gap between ML development and production. Its ability to scale, distribute computing, and support various frameworks such as XGBoost, PyTorch, and TensorFlow makes it an attractive choice for developers building ML platforms and components.

Ray Serve is an online model inference framework that focuses on elastic scaling, optimizing inference graphs, and supporting multiple machine learning frameworks, including Java. It aims to provide a universal API for distributed computing by offering simple yet general programming abstractions. Ray Serve enables developers to compose multiple libraries within a single distributed application, making it an excellent "distributed glue" system. The framework supports Java natively, allowing users to deploy their own Java code and call and manage it through the Java API. Users can create, query, and modify deployments using the Java API, which also allows for cross-language functionality between Python and Java.

This blog post from Ant Group discusses their implementation of scalable Ray Serving architecture atop Ray, deploying 240,000 cores for model serving, scaling by 3.5x from the previous year, and reaching 1.37 million TPS during peak times. The authors introduce Ant Ray Serving as an online service platform that can deploy users' Java and Python code as distributed online services, providing scaling, traffic routing, and monitoring capabilities. They highlight two business scenarios: Model Serving and EventBus, which is an event-driven serverless platform based on Ant Ray Serving. The authors discuss the challenges they faced with model serving, including performance and traffic isolation issues, and how they addressed them using Ray Serving's service isolation and resource isolation features. They also introduce automatic scaling capabilities for both service instances and Ray clusters. Additionally, they mention their cooperation with the Anyscale Ray Serve team to integrate Ant Ray Serving into the open source Ray Serve architecture, providing support for Java language in Ray Serve, cross-language deployment, and componentization capabilities with flexible scalability.