September 2024 Summaries

Younes Abouelnagah, a Principal ML Engineer at Roblox, shares how his team scaled their online NLP ML model inference on CPU machines and reduced latency using Ray, a distributed computing framework for Python. The blog post details the process of scaling up and out, reducing latency and CPU usage while maintaining civility on the platform by running user-generated content through multiple models. It highlights key learnings in using Ray Core to scale the serving of ML models with very low latency requirements, including setting up a dedicated Ray cluster for improved performance and efficiency.

This blog showcases the versatility of Ray, an open-source unified compute framework, by demonstrating embedding generation and LLM batch inference with Ray in two Flyte pipelines. Flyte is an open-source orchestrator that facilitates building production-grade data and machine learning pipelines. The blog also highlights the importance of a unified distributed computation framework like Ray and a workflow orchestrator like Flyte for managing AI/ML workloads. Anyscale, built by the creators of Ray, provides a seamless user experience for developers to deploy AI/ML workloads at scale, while Union, built by the technical founding team behind Flyte, abstracts away the infrastructure, providing a turnkey system that lets ML engineers and data scientists focus on their tasks. The blog then dives into two Flyte pipelines: one for generating embeddings using Ray Data and saving them to cloud storage shared by Union and Anyscale; and another for monitoring GitHub issues in Flyte repositories and using the Anyscale Platform to serve an LLM with RAG to perform batch inference and reply to the GitHub issues.

This blog post provides a comprehensive guide on fine-tuning large language models (LLMs) such as Llama-3, Mistral, and Mixtral using Anyscale. It covers the entire process from preparing input data to launching the fine-tuning job and monitoring the process. The article also discusses serving your model with Anyscale's ray-llm library, including how to serve both LoRA and full-parameter fine-tuned models. Additionally, it offers tips on optimizing for compute cost and monitoring the training progress.

This blog post explores scaling up a pipeline that generates text embeddings using Ray Data and Sentence Transformers. The author demonstrates an easy migration from a pandas-based pipeline to a Ray Data-based pipeline, highlighting significant performance improvements with minimal code changes. The improved Ray Data pipeline delivers a 10x performance improvement over the naive implementation and allows for distribution of workload across a cluster of machines with GPUs and CPUs compared to running pandas on a single machine.