June 2025 Summaries

As unstructured data like text, images, and videos grow exponentially, traditional data and AI infrastructures, which focus on structured data and SQL-style workloads, face new challenges. Python-based AI models have also expanded beyond the capabilities of non-Python distributed engines, creating bottlenecks in AI production. Anyscale addresses these issues with Ray, a distributed compute framework designed for Python, multimodal data, and heterogeneous compute environments. Since its release in 2017, Ray has supported large-scale AI applications, such as OpenAI's GPT-3.5 and Ant Group's production model-serving systems, by integrating GPU and CPU workloads efficiently. It complements existing AI frameworks by simplifying distributed computing tasks like task scheduling, data movement, and autoscaling, enabling teams to build cost-effective, scalable infrastructure for complex AI cases. This shift is essential as AI evolves beyond SQL and batch processing, demanding new tools for the multimodal AI era.

Ray is an open-source distributed computing engine that precisely orchestrates infrastructure for any distributed workload on any accelerator at any scale. It consists of three layers: Ray Core, Ray AI Libraries, and Ray Deployment. Ray Core provides a small set of essential primitives (task, actor, and object) for building and scaling distributed applications, allowing users to program distributed applications with the Ray Core API just like programming on a laptop. The ecosystem includes various AI libraries such as Ray Data, Ray Train, Ray Tune, Ray Serve, and RLlib that cover the ML lifecycle, from data processing to training to tuning to serving. KubeRay is a Ray Kubernetes operator that simplifies the management of the lifecycle of Ray clusters and associated applications on Kubernetes, enabling data scientists and ML scientists to focus on their machine learning logic while infra engineers concentrate on Kubernetes. ACK (Alibaba's Container Service for Kubernetes) supports KubeRay as a managed component, offering advantages such as elastic compute, observability, security, zero operations and maintenance, high availability deployment, and resource policy API to orchestrate compute resource types by defining priorities of node preferences. The native Ray Dashboard is available only while Ray clusters are running, but ACK provides the Ray History Server for access to dashboards for both active and terminated RayCluster custom resources. By leveraging kube-queue integration and the Ray History Server, users can successfully operationalize Ray workloads in production while utilizing resources efficiently.

The software stack for AI compute consists of three layers: the training and inference framework, the distributed compute engine, and the container orchestrator. The training and inference framework includes PyTorch, vLLM, and other frameworks designed for model parallelism and transformer-specific optimization. The distributed compute engine, such as Ray, handles scheduling, data movement, and failure handling. The container orchestrator, like Kubernetes or SLURM, allocates resources and manages the lifecycle of containers. This stack is used by various companies, including Pinterest, Uber, Roblox, and others, to manage AI workloads, including training, inference, and batch processing. Post-training frameworks, such as VeRL, SkyRL, OpenRLHF, Open-Instruct, and NeMo-RL, are also built using this stack, often combining Ray, PyTorch, vLLM, and other technologies.

This blog builds upon our previous guide to RAG-based applications, providing a deeper look at real-world challenges and showcasing how Anyscale and Ray can help build more scalable, production-ready Retrieval-Augmented Generation systems. The comprehensive series of notebooks guides users through the basics and enables them to build their own real-world solutions. By leveraging distributed computing with Ray and Anyscale's managed, reliable clusters, enterprises can unlock value from unstructured documents and reduce hallucinations, provide transparent citations, and incorporate new information without model retraining.