February 2022 Summaries

The Emirates Team New Zealand has won the America's Cup for the second time in four years, thanks to the help of a new AI agent developed by McKinsey's QuantumBlack using Ray and RLlib. The AI agent was trained on simulations of various sailing scenarios, allowing it to learn from trial and error and refine its performance over time. This enabled the team to iterate through design iterations much faster than humans, compressing hours into days. The use of reinforcement learning and Python framework built on Ray has successfully accelerated the design process by a factor of 10 for complex problems like sailing hydrofoil designs.

The text discusses the mathematical framework for designing Reinforcement Learning (RL) algorithms. It begins by defining the problem setting, including the observation space, action space, reward function, and terminal condition. The goal of RL algorithms is to learn a policy that maximizes the total rewards within an episode, but there is a chicken-and-egg problem since the agent needs to see future returns to act on the environment. To overcome this, RL agents form an estimate of what they think will be in the future if they act according to their current policy, leading to the concept of expectation. The main goal of RL algorithms is to learn a policy that achieves the maximum expected returns in its environment. The RL loop involves observing the world, acting on it, and receiving a reward, with the agent updating its policy using collected data to maximize rewards. A discount factor is introduced to prioritize near-term rewards and set the length of the time horizon for maximizing rewards. The framework is very general, allowing for the design of various RL algorithms without making assumptions about the underlying problem.

Deploying a machine learning model can be challenging depending on the deployment platform and tools used to serve it into production, so selecting the proper tools and best platform is crucial for effective serving and deployment of models. Various frameworks such as TorchServe, Flask, FastAPI, and Ray Serve are available for serving PyTorch models, each with its pros and cons. TorchServe is a flexible tool developed by PyTorch for serving PyTorch and Torch-scripted models but has drawbacks like frequent updates and being Java-dependent. Cloud-hosted solutions like Amazon SageMaker can be powerful but expensive, while web-based frameworks like Flask are efficient but may present scaling challenges. Ray Serve, on the other hand, is a library that runs on top of the Ray Distributed Library Ecosystem, providing a simple web server for production deployments with end-to-end control over the request lifecycle and allowing models to scale independently. The integration of FastAPI and Ray Serve can help in scaling PyTorch model serving API across a Ray cluster, improving the number of requests served per second and reducing latency. With this setup, deploying and serving scalable machine learning models into production becomes more manageable, offering various tools and frameworks to meet different needs.

This series on reinforcement learning was guest-authored by Misha Laskin while he was at UC Berkeley. Misha's focus areas are unsupervised learning and reinforcement learning. Reinforcement learning (RL) has played a critical role in the rapid pace of AI advances over the last decade, allowing algorithms to both interact with the world and reason over periods of time to achieve goals. RL algorithms work similarly to training dogs to learn new tricks, receiving numerical rewards for desired actions, but face challenges such as the exploration problem, where they must balance finding optimal solutions with exploring their environment. The ideas that have shaped modern RL were formalized in 1989 by Chris Watkins and later popularized by breakthroughs like AlexNet and DeepMind's Deep Q Network (DQN), which paved the way for advances in image classification, video game playing, and other areas. Today, RL is being applied to solve a variety of practical problems, including recommender systems, autonomous vehicle navigation, and data center cooling, with algorithms tailored to specific applications like SlateQ for product recommendations and Google's new algorithm addressing implicit bias.

XGBoost is a popular open-source implementation of the gradient boosting algorithm, known for its efficiency and model performance. To speed up XGBoost model training, three approaches can be used: the tree method (using GPU acceleration), cloud training using services like AWS or Google Cloud, and distributed XGBoost-Ray on Ray, which leverages multiple nodes and GPUs to achieve significant improvements in training time. The latter approach is found to be the most effective way to reduce training time, thanks to its multi-node training capabilities, full CPU support, full GPU support, and configurable parameters like RayParams.

The hyperparameter tuning process allows machine learning models to find the optimal settings for the best results from our model. Hyperparameter tuning can be time-consuming and tedious, but automated methods like grid search, random search, and Bayesian optimization can help us find optimal hyperparameters more quickly. Ray Tune is a distributed hyperparameter tuning library that can accelerate the process by trying different combinations of hyperparameters in parallel on multiple computers. We can use Ray to distribute our hyperparameter tuning across a cluster, which can significantly reduce the time needed to tune our model. By using Ray and Ray Tune, we can rapidly find optimal hyperparameters for our machine learning models, such as the digit identification model, without requiring significant code refactoring. The process allows us to efficiently improve our model's performance by distributing the work across multiple nodes in the cluster.

Ray Datasets is a data loading and preprocessing library built on top of the Ray framework, designed to simplify machine learning (ML) pipelines by providing a flexible and scalable API for working with data within Ray. It leverages Ray's task, actor, and object APIs to enable large-scale ML ingest, training, and inference, all within a single Python application. Datasets supports popular storage backends and file formats, common ML preprocessing operations, and works seamlessly with Ray-integrated libraries and ML frameworks such as TensorFlow and Torch. The library aims to be a universal parallel data loader, providing a narrow data waist for Ray applications and libraries to interface with. It offers convenient data preprocessing functionality, supports running stateful computations on GPUs, and enables batch inference on large datasets. With its robust distributed dataplane, Datasets delegates most of the heavy lifting to the Ray dataplane, focusing on higher-level features such as convenient APIs, data format support, and stage pipelining.

This blog post is part 2 in a series on hyperparameter tuning, focusing on practical examples with XGBoost and the MNIST dataset. The authors provide step-by-step instructions on how to preprocess the data, build a model without hyperparameter tuning, tune hyperparameters using random search, and demonstrate improvements in accuracy. They also outline their plans for part 3, which will explore distributed hyperparameter tuning using Ray Tune.

This blog post is part 1 in our series on hyperparameter tuning, exploring the importance of controlling machine learning model behavior through optimal hyperparameter settings, which can significantly impact model performance and accuracy. Hyperparameter tuning involves finding the best combination of parameters that maximize model performance, minimizing errors, and ensuring reliable results across various data sets. The article discusses key concepts, including differentiating between parameters and hyperparameters, understanding the role of hyperparameters in algorithmic learning, and the importance of tuning these settings to achieve optimal results. It also delves into practical examples of hyperparameters for specific models such as neural networks and XGBoost, highlighting the need for careful consideration when selecting values. The article concludes by introducing three primary methods for hyperparameter tuning: grid search, random search, and Bayesian optimization, each with its strengths and limitations, and emphasizes the importance of choosing an approach that balances computational efficiency with optimal results.

Ray version 1.10 is now available, featuring Windows support in beta, enhancements to job submission including log streaming and custom headers for authentication, allowing users to access these features with a simple pip install -U ray command.

In 1949, The Star newspaper predicted that computers would one day help with income-tax calculations, but this was considered speculation at the time. Today, top-of-the-line laptops can handle astronomical calculations at lightning speed, a fact that has become reality. To deploy machine learning models in production, developers need to consider ease of development and tuning/training at scale. Python is the de-facto programming language for machine learning workloads, thanks to its PyData ecosystem and ease of use. Developers prefer to customize their environments using conda or virtual environments to match their staging or production environment. To achieve this setup, two distinct environments can be created with required dependencies, allowing developers to seamlessly switch between them. When model training requires more compute-intensive resources than a laptop can offer, an existing cluster can be extended to meet those requirements. The Ray.io platform is recommended for three reasons: it provides a simple client to connect to an existing cluster, leverages native libraries for ML-specific tasks, and allows developers to use their favorite IDE or development tools while still scaling up to a cluster. Ray Tune is used for tuning and training at scale, providing state-of-the-art algorithms and easy-to-use APIs. It can be used with popular tuning libraries like Hyperopt and Optuna, giving developers the tools and techniques to do model training and tuning at scale. Ray Train allows for distributed data-parallel deep learning training using PyTorch or TensorFlow models, offering composability and interoperability with Ray Tune. Both Ray Tune and Ray Train are interoperable, allowing developers to use them together when needed. The MLflow platform is recommended for tracking experiments and results, providing a fluent API and robust integrations with both Ray Tune and Ray Train. Developers can log metrics, parameters, and artifacts using callbacks like MLflowLoggerCallback, making it easy to view results in the MLflow UI. By following these considerations, developers can ensure their machine learning models are deployed, served, and inferred at scale.

I'm thrilled to be joining Anyscale, a company that has the potential to revolutionize distributed computing by building upon my past experiences with Ray and MapReduce. I've seen firsthand the value of Ray in real-world production situations, particularly in the AI space, and I'm excited to contribute to making it even more accessible and robust. The team at Anyscale is talented, humble, and fun-loving, and I look forward to working alongside them to bring my expertise to bear on solving big problems. With a strong foundation and ambitious long-term vision, I believe Anyscale has the potential to make a significant impact in the software engineering community.