October 2024 Summaries

The collaboration between Anyscale and Astronomer aims to provide a comprehensive solution for orchestrating and scaling machine learning and AI workflows. By combining Airflow's powerful workflow management with Ray's distributed computing capabilities, organizations can efficiently scale, integrate, and automate complex machine learning workflows, streamlining the entire data lifecycle from processing to model deployment. The integration of these two platforms offers seamless scalability, end-to-end visibility, optimized resource utilization, and accelerated development and deployment of ML models.

Compiled Graphs is a new feature in Ray that offers significant improvements for large AI model workloads, such as training and inference. Unlike traditional CPU-based workloads, these tasks are overwhelmingly GPU-intensive and often require distributed computation across multiple accelerators. Compiled Graphs provide minimal task submission overhead compared to Ray's standard task submission overheads, enabling faster execution of sub-second workloads like auto-regressive token generation. Additionally, it supports native GPU to GPU transfer while automatically resolving deadlock and overlapping communication with computation. These improvements open up exciting new opportunities for Ray programs, such as reduced system overhead for repetitive task graphs, native support for GPU-GPU communication via NVIDIA NCCL, and optimized scheduling to avoid deadlock and best utilize compute and communication resources.

Ray Summit 2024 highlighted the growing demand for scalable, flexible, and powerful AI infrastructure as Ray now orchestrates over 1 million clusters per month. However, companies are facing an AI Complexity Wall due to increased models, data modalities, and accelerators. To overcome this challenge, businesses need an AI Compute Engine like Ray that supports various workloads, decouples resource allocation, and abstracts away infrastructure complexity. Key features of Ray include the new compiled graphs API for faster GPU communication, Ray Data for unstructured data preprocessing, and scalability improvements to support up to 8,000 nodes. Anyscale, built by Ray creators, offers additional enhancements in performance, efficiency, flexibility, and governance.

Ray, a popular framework for large-scale data processing workloads, is being improved by Anyscale to enhance performance and reliability. The new accelerated metadata fetching feature in Anyscale can reduce start-up time by up to 4.5 times compared to open-source Ray for a 1 TiB test dataset containing 128 MiB files. This enhancement leads to faster development cycles, more efficient use of compute resources, and reduced wasted cycles. Anyscale's optimized version of Ray, RayTurbo, demonstrates significant speed improvements in start-up time and overall data processing efficiency when compared with open-source Ray. These enhancements are available on the Anyscale platform for all users without additional configuration requirements.

Anyscale has expanded its capabilities with Kubernetes support, allowing users to integrate distributed AI workloads into their existing Kubernetes-managed infrastructure. This update addresses the growing demand for flexibility, control, and seamless integration with the tools and environments users already have in place for machine learning and AI tasks. The new Anyscale Operator for Kubernetes provides access to high-performance unified AI compute platform, offering enhanced security, governance, observability, cost optimization, custom pod definitions, integrated monitoring and logging, and compatibility with almost all Kubernetes clusters. Key launch partners include Amazon EKS, Google GKE, Azure AKS, and Oracle Cloud Infrastructure Kubernetes (OKE).

Anyscale, a distributed computing platform for Ray, is now available on AWS Marketplace, offering seamless integration with Amazon Web Services (AWS) ecosystem. This collaboration enables users to leverage optimized Ray performance, cost efficiency, developer tooling, and enterprise governance features. Additionally, Anyscale has achieved AWS Generative AI Competency and is featured in the AWS GenAI Marketplace, showcasing its leadership in AI infrastructure. By using Anyscale on AWS, organizations can simplify billing, streamline procurement, access flexible pricing terms, and enhance governance for their AI/ML and Generative AI workloads.

Large Language Models (LLMs) have revolutionized the technology industry, with a focus on optimizing inference costs due to high GPU prices. While online inference provides low-latency responses, batch inference for LLMs offers higher throughput and greater cost-effectiveness by optimizing GPU resource utilization. In certain cases, Anyscale can reduce costs by up to 2.9x compared to online inference providers such as AWS Bedrock and OpenAI. RayLLM-Batch is a library leveraging Ray and Anyscale components to optimize LLM batch inference at scale, offering a powerful, cost-effective solution for large-scale batch LLM inference. Experiments show that the Anyscale FP8 batch inference solution can outperform other common solutions on price-performance.

Ray Data, a scalable data processing library for AI workloads, has been announced as generally available with stability improvements in streaming execution, reading and writing data, better tasks concurrency control, and debuggability improvement with dashboard, logging and metrics visualization. It is built to efficiently use mixed CPU and GPU resources, integrates seamlessly with both the data and AI ecosystems, while offering fault tolerance and resource multiplexing properties of traditional batch processing systems. Ray Data supports unstructured data preprocessing, batch inference, and ingest for ML training workloads. It also integrates with most common tools between the data and AI ecosystems such as ML frameworks, file formats, data sources, and cloud storage.

Anyscale Workspaces is an out-of-the-box solution for developers, providing advanced observability features to monitor and track development workflows. It simplifies dependency management, enhances log viewing, and introduces serverless mode. The platform enables data scientists and machine learning engineers to build distributed apps on large scale clusters with ease. Anyscale Workspaces is designed to improve the developer experience by making it faster and simpler to develop and deploy applications.

Anyscale is now available on Google Cloud Platform (GCP) Marketplace, offering enhanced performance, scalability, reliability, and developer tooling for Ray's AI Compute Engine. By leveraging Anyscale on GCP, organizations can build and scale AI/ML models more efficiently. The platform provides optimized Ray performance, cost efficiency, developer tooling, enterprise governance, expert support, and seamless integration with existing investments on GCP.

Efficiency is crucial for AI applications, both in development and production. However, a common experience among AI practitioners is spending significant time waiting for instances to boot, containers to pull, and models to load. Anyscale has optimized scale-up speed across the entire stack, leading to up to 5.1x faster autoscaling for Meta-Llama-3-70B-Instruct on the Anyscale platform compared to running the same application using KubeRay on Amazon Elastic Kubernetes Service (EKS). Faster scale-up speeds benefit AI engineers and researchers by enabling quick iteration, avoiding idle time in development, and autoscaling to meet workloads' demands while avoiding idle resources in production. The Anyscale Platform provides a fully-managed Ray solution with tailored infrastructure for high performance, cost effectiveness, and fast model loading.

Anyscale introduces new enterprise governance and observability tools aimed at helping organizations better control their AI infrastructure, understand utilization, and improve efficiency of ML workloads. These tools provide comprehensive insights into compute resources usage, allowing customers to optimize application performance and resource allocation across clouds and compute resources. Key features include cluster-wide visibility, spot utilization insights, and Ray telemetry. Additionally, Anyscale offers Resource Quotas for granular control over cloud resource allocation and usage, ensuring cost control, fair allocation, transparency, and customizability to meet unique organizational needs.

Anyscale has introduced RayTurbo, an optimized runtime for Ray on its platform. The new offering aims to provide the best price-performance and developer capabilities for AI workloads compared with other solutions including running Ray in open source. Among other optimizations, RayTurbo reduces runtime duration of read-intensive data workloads by up to 4.5x compared to open source Ray on certain workloads, accelerates end-to-end scale-up time for Llama-3-70B by up to 4.5x compared to open-source Ray on certain workloads, and reduces LLM batch inference costs by up to 6x compared to repurposed online inference providers such AWS Bedrock and OpenAI. The platform is focused on four broad workloads in the AI development lifecycle: data processing, training, serving, and LLM workloads.