November 2022 Summaries

The large scale machine learning workloads on Kubernetes often suffer from a lack of resource reservation systems, leading to gang scheduling issues where jobs are stuck waiting for resources to become available. KubeRay with the Multi-Cluster-App-Dispatcher (MCAD) controller helps to avoid such situations by queuing each Ray workload until resource availability requirements are met. MCAD allows users to queue each of their Ray workloads until aggregated resources are available in one of the Kubernetes clusters, ensuring that all pods can be scheduled. With KubeRay and MCAD, users can scale their Python and AI applications from a laptop to a cluster seamlessly, using gang scheduling and workload pre-emption capabilities. The Multi-Cluster-App-Dispatcher (MCAD) is a Kubernetes controller providing mechanisms for applications to manage batch jobs in a single Kubernetes cluster or multi-Kubernetes-cluster environment, allowing users to queue Ray clusters until resource availability requirements are met.

Biolexis Therapeutics, a preclinical-stage biopharmaceutical company, is leveraging AI and machine learning to develop small-molecule therapeutics targeting various diseases for precision medicine. The company's FIELDS platform uses advanced AI/ML techniques to generate accurate protein structure models, providing a deeper understanding of proteins' functions and their effects on cellular structures. By utilizing the Anyscale Platform and Ray, Biolexis can scale its workload and accelerate scientific outcomes, reducing costs and increasing the probability of success in clinical trials. The platform enables seamless scaling of workloads, parallelization of processes, and simplified deployment to production without requiring a dedicated infrastructure team.

Richard Decal, Lead ML Engineer at Dendra Systems, is using machine learning to combat climate change by scaling ecosystem-restoring drones globally. Traditional ecological restoration methods are not scalable and often fail in certain terrains. Dendra Systems aims to enable faster, high-quality, transparent, and scalable restoration through the use of ultra-high resolution mapping with drones, machine learning algorithms for analysis, and specialized seeding drones. Their approach is 11 times faster and three times cheaper than traditional methods. Ray ecosystem and Anyscale platform have been instrumental in building their machine learning platform capable of handling large image datasets. The technology allows them to offer a suite of ecosystem analytics services such as biodiversity assessments, weed management, pest management, erosion and soil health monitoring, and more at a massive scale.

Wildlife Studios, a leading mobile gaming company, faced challenges with their legacy system for serving revenue-generating in-game offers due to latency and cost requirements. They switched to Ray Serve on Anyscale, which allowed them to serve offers three times faster. This improved gaming experience led to increased revenue from more relevant and faster served in-game offers while reducing infrastructure costs through better CPU utilization. The company's Dynamic Offers team leveraged the benefits of Ray Serve's architecture for parallel inference on multiple models, resulting in decreased latency and minimized idle machines.

The Adtriba company has developed a marketing measurement system that uses AI to help businesses optimize budget allocation. Their system is based on three pillars: Advanced Multi-Touch Attribution, Media Mix Modeling, and Experiments. The company's approach to media mix modeling requires the use of sophisticated AI/ML models that can address challenges such as carry-over effects, shape effects, and changing media efficiency over time. To overcome these challenges, Adtriba uses non-linear optimization, Bayesian inference, and hyperparameter optimization methods. They have developed custom Media Mix Modeling packages using various frameworks and tools, including Ubers Orbit, Numpyro, and scikit-learn. The Anyscale Platform, built on Ray's multi-objective hyperparameter search algorithms, provides an advantage in terms of scalability, calibration capabilities, and ability to find the pareto-optimal combination of hyperparameters that optimizes model fit and attribution accuracy. With this platform, Adtriba is able to run thousands of iterations quickly and efficiently, reducing the time required to optimize media mix models from a week to under 3 hours.

Ray 2.1 introduces a new feature that displays native time series metrics as part of its dashboard, allowing users to view insights into the scheduling and performance of their Ray workloads. This feature provides easy integration with Prometheus and Grafana, enabling users to monitor and debug their production environments more effectively. The dashboard now displays time series graphs for critical system states such as scheduler slot usage, CPU/GPU utilization, memory usage, and task states over time. Additionally, the Ray 2.1 release exposes time series charts for both physical and logical resource slot usage, providing users with a better understanding of cluster utilization. This new feature addresses the observability imperative in distributed systems, enabling users to monitor and debug their Ray production workloads more efficiently.

Batch inference in Ray can be implemented using low-level primitives such as tasks, actors, or actor pools, or high-level APIs like BatchPredictor. The choice depends on the desired level of control and complexity. With low-level primitives, control is given over how to execute batch inference, but requires understanding of Ray's core primitives and implementation details. In contrast, BatchPredictor provides a more declarative and expressive API for batch inference, offering automatic scaling and less code. For data scientists and machine learning practitioners who prioritize scalability and ease of use, BatchPredictor is a desirable option.