April 2023 Summaries

The market opportunity in AI infrastructure is enormous and growing quickly, with cloud hyperscalers investing heavily in specialized chipsets and platforms for AI applications. Anyscale is well-positioned to solve the challenges of AI infrastructure, having built a thriving open source community around its Ray project, which drives product-market fit and velocity. The company has also attracted top talent from academia and industry, with a "win as a team" culture that fosters collaboration and innovation. As the AI landscape continues to evolve, Anyscale is poised to play a significant role in powering sophisticated use cases, and the author is excited to be a part of the team as Head of Marketing. With the potential for AI applications and use cases to revolutionize industries, Anyscale's infrastructure will be crucial to unlocking their full potential.

The Ray 2.4 release features exciting improvements across the Ray ecosystem, including enhancements to Ray data for ease of use, stability, and observability, improved Serve observability, introduction of RLlib's module for custom reinforcement learning, improved Ray scalability for large clusters, new examples for Generative AI workloads such as Stable Diffusion and LLMs like GPT-J, and the introduction of a new LightningTrainer to scale PyTorch Lightning on Ray. The release aims to make Ray a pivotal compute substrate for generative AI workloads and address challenges associated with open-source generative AI infrastructure. With this update, users can now use Ray with Stable Diffusion and LLMs like GPT-J, fine-tune these models using DeepSpeed and Hugging Face, and build an open-source search engine with Ray and LangChain. The release also introduces a new RLModule abstraction in RLlib to define custom reinforcement learning models, improved Serve observability, and support for larger scale workloads up to 2000 nodes.

I've worked on cutting-edge problems across various technological transitions, including the World Wide Web, Mobile, and Cloud computing, and have contributed to widely-adopted open-source projects such as Wireshark, Nginx, and Ray. I've helped companies overcome infrastructure obstacles while empowering them to benefit from advancements in AI and related technologies. Joining Anyscale allows me to tackle the complexities of AI and make it available to all organizations, providing better tools and doing so responsibly. The company's technology is proven and rapidly evolving with strong community involvement, offering vast opportunities for development and deployment of AI solutions. I'm now leading product at Anyscale, which aims to shape the future of AI and its impact on our lives.

This blog series introduces LangChain and Ray Serve, two powerful tools for building a search engine using LLM embeddings and a vector database. LangChain provides an amazing suite of tools for everything around LLMs, including indexing, generating, and summarizing text, while Ray Serve makes it easy to deploy a LangChain service in the cloud. The blog series will show how to build a store, speed up indexing by parallelizing embedding, serve the search results, and enable request batching. It also covers scalability and cost, and will share Part 2 and Part 3 of the series where they will discuss turning this into a chatgpt-like answering system and talk about scalability and cost respectively. The code for this is available in a Github repo, and there are resources available to learn more about Ray, including a hosted service for ML Training and Serving.

LiveEO, a German startup, partnered with Anyscale and Prefect Cloud to accelerate and optimize their geospatial workloads, achieving a 65% reduction in processing time using Python-based solutions for earth observation data analysis. LiveEO's data science team upgraded their ML stack to standardize, automate, and version pipeline updates, releasing packages via API or UI and abstracting infrastructure for seamless scaling of CPU and GPUs. With Anyscale and Prefect Cloud, LiveEO reduced runtime by 30% and increased continuous delivery, resulting in faster and more reliable customer service and a significant boost to their bottom line.

This blog post discusses the use of Ray, HuggingFace, DeepSpeed, and PyTorch to build a system for fine-tuning and serving Large Language Models (LLMs) in a cost-effective and efficient manner. It highlights the benefits of using this tech stack, including its simplicity, speed, and scalability. The authors demonstrate how to fine-tune a 6 billion parameter GPT-J model on Shakespeare's works and serve it as a web service using Ray and HuggingFace. They also discuss the importance of cost-effectiveness in LLM applications, particularly when dealing with large models and high-performance computing requirements. By leveraging Ray's distributed capabilities, the authors show that running multiple machines can be both cheaper and faster than using a single large machine.