November 2020 Summaries

Reinforcement learning (RL) algorithms are often brittle to hyperparameter choices, making automation of hyperparameter discovery essential. To address this, researchers have developed Population-Based Bandits (PB2), a new method for training neural networks that uses a probabilistic model to guide the search in an efficient way. PB2 is based on ideas from GP-bandit optimization literature and combines the benefits of Population-Based Training (PBT) and Bayesian Optimization. It leverages a Gaussian Process model to adapt its model hyperparameters during training, allowing it to efficiently find high-performing hyperparameter configurations with fewer agents than required by prior work. PB2 has been open-sourced in Ray Tune v1.0.1, enabling users to try it out themselves and explore its potential for efficient hyperparameter tuning in reinforcement learning tasks.

The history of computing infrastructure has seen steady improvements over time, with cloud computing introducing concepts such as elimination of upfront hardware investments and elasticity. In 2015, AWS introduced Lambda, a serverless computing service that abstracts away infrastructure and provides finer-grained billing increments compared to traditional cloud computing. Since then, interest in serverless and cloud functions has grown, with all major cloud providers now offering similar services. However, current cloud functions have limitations such as lack of support for stateful applications, direct communication between tasks, access to hardware accelerators, and fine-grained coordination and control. The distributed computing framework Ray addresses many of these challenges and is considered the right foundation for a general purpose serverless framework.

The Hugging Face transformers library has become critical in natural language processing due to its cutting-edge research implementations, thousands of trained models, and accessibility. Simple experiments have shown the benefit of using advanced tuning techniques like genetic optimization for large performance improvements over standard hyperparameter optimization methods. The Transformers 3.1 release integrated Ray Tune, a popular Python library for hyperparameter tuning, providing a simple yet powerful integration that allows users to easily access powerful hyperparameter tuning solutions without sacrificing customizability. This integration enables users to leverage state-of-the-art algorithms and tooling like Weights and Biases and tensorboard, making it easy to fine-tune models on various datasets, including MRPC. The Hugging Face + Ray Tune integration is demonstrated through an example that uses HyperOptSearch for hyperparameter search with optional usage of Weights and Biases.