March 2022 Summaries

The PyTorch team has introduced a new Ray Scheduler for TorchX, which allows developers to run scalable and distributed PyTorch workloads without setting up infrastructure or changing training scripts. This scheduler is built on top of the Ray framework, providing a rich set of native libraries for ML workloads and a general-purpose core for building distributed applications. With TorchX's Ray Scheduler, users can easily deploy their PyTorch machine learning applications from R&D to production, leveraging the ecosystem of libraries and integrations available in PyTorch Distributed and PyTorch Lightning. The scheduler provides features such as hyperparameter optimization, model serving, and distributed data-parallel computing, allowing developers to build complex pipelines while decoupling their training script from infrastructure. Users can submit jobs to a cloud of their choice using the TorchX CLI or SDK, and monitor job status and progress through the TorchX SDK and Ray Jobs API. The joint engineering effort between Meta AI PyTorch and Anyscale ML teams has made this feature possible, enabling users to run scalable and distributed PyTorch workloads with ease.

Ray Summit is a two-day in-person conference for the Ray community, taking place later this year (August 23-24), with a call for papers submission deadline of April 18. Speaking at Ray Summit offers benefits such as sharing expertise and knowledge, building public speaking skills, and gaining recognition, while also providing free access to the event. The conference covers various themes and topics, including scalable ML, reinforcement learning, and cloud computing, with three types of talks: technical lightning talks, technical talks, and technical deep dives. To submit a successful proposal, focus on a few key areas, keep it simple and straightforward, edit thoroughly, and consider your audience's needs, with tips available on the Ray Summit page.

The upcoming Production RL Summit aims to bring together machine learning engineers, data scientists, and researchers to share knowledge and learn from industry experts in reinforcement learning (RL). The event will feature a diverse lineup of speakers, including those from J.P. Morgan, Siemens, and Riot Games, who will discuss various applications of RL across industries such as finance, gaming, and manufacturing. The summit will provide hands-on learning opportunities, networking sessions, and success stories to help attendees build practical RL skills. Experts will share insights on how RL can be used in production environments, address limitations with offline RL, and deploy advanced agents into industrial settings. The event is free and virtual, making it accessible to a wide range of participants, from novices to experts.

Redis was used as a metadata store and pub/sub broker in Ray until version 1.11.0, but starting from Ray 1.11, Redis is no longer the default. The changes allow Ray to focus on adding better support for fault tolerance and high availability in the future. In the pre-1.0 Ray architecture, Redis was used as a backing store for key-value storage and message pub/sub. However, this approach had several shortcomings, including finite CPU and memory limitations, inefficient use of Redis pub/sub, and increased coupling between components with direct access to Redis. The new architecture in Ray 1.11 eliminates these issues by storing cluster metadata inside the Global Control Store (GCS) instead of Redis and using internal broadcast/pub/sub implementations for actor/node/worker state updates. Although Redis is no longer stored as the default, it can still be used as an external metadata store. The updated configuration process also changes to use the RAY_REDIS_ADDRESS environment variable.

The latest version of Ray, a distributed computing framework, has been released, marking an important step in its evolution by removing the default Redis dependency, opening the door to better support for fault tolerance and high availability in future releases. The new release also introduces a more intuitive documentation structure, organized around three primary use cases: Ray ML, Ray Core, and Ray Clusters. Additionally, Ray is now stable for Python 3.9, allowing users to run it in production with confidence. With these changes, Ray aims to improve its performance, scalability, and usability, providing a better experience for developers and researchers alike.

This series on reinforcement learning discusses two important limitations that can make Q learning unstable: the Bellman error and optimism. The first problem, the Bellman error, occurs when the target prediction is also a prediction, causing a runaway loss due to changes in both the current state and next state values. A practical solution to this issue is to use two different but similar networks to predict both the current and next state values, such as the original DQN's approach of keeping an old copy of the Q network or maintaining a target network with exponentially moving average weights. The second problem, optimism, arises from the Bellman equation and causes predictions made by Q functions optimized via the Bellman error to be too optimistic, leading to overestimation of future returns. To address this issue, double Q networks are proposed, which train two independent Q functions that select the minimum value as the target prediction, resulting in more conservative estimates of the Q function than DQN. Understanding these limitations and solutions allows intuition for RL systems and development of practical solutions to overcome common challenges when training RL algorithms.

XGBoost is a highly efficient and flexible gradient boosting library designed to provide fast and efficient solutions for several data science and machine learning problems. It has quickly become the state-of-the-art machine learning algorithm for solving tasks with structured data due to its high speed and exceptional performance. Options for serving XGBoost models include cloud-hosted platforms, but these can be expensive and only work within their own ecosystems. Manually deploying machine learning models from concept to production is complex and time-consuming, so several frameworks are used to deploy XGBoost in production. In this article, two frameworks, Flask and Ray Serve, are highlighted as options for deploying XGBoost models. Flask is considered an exceptional deployment framework due to its ease of setup, efficiency with REST endpoints, and lack of dependencies on external libraries. However, it has drawbacks such as being unsuitable for large applications and lacking login and authentication capabilities. Ray Serve provides a simple web server that leverages complex routing, scaling, and testing logic necessary for production deployments, making it easier to scale out models on a multi-node cluster without changing the code. With Ray Serve, XGBoost models can be deployed successfully with just a few lines of code, allowing for high-efficiency and high-performance production deployments.

This series on reinforcement learning explores the concept of Q functions and their application in Q learning algorithms. The goal of RL algorithms is to learn a policy that achieves maximum expected returns in its environment. A Q function predicts how much return an agent expects to get if it takes a specific action, and the agent's goal is to achieve this value. The Bellman error is used as a loss function for RL, which can be computed using a neural network. The Q learning algorithm involves training an agent to minimize the Bellman error by sampling transitions from a replay buffer and choosing actions based on epsilon greedy strategy. This simple algorithm has been used in breakthroughs like Deep Q Networks and is a foundation for other algorithms in the field of Deep RL.