May 2022 Summaries

As part of their community-building efforts, the Anyscale team embarked on a Saturday morning litter-picking initiative, collecting hundreds of pieces of trash from their neighborhood. This volunteer activity not only helped keep their community clean but also fostered team bonding and camaraderie among colleagues. Anyscale's emphasis on community is reflected in their Volunteer Time Off (VTO) policy, which encourages employees to participate in local volunteer work while promoting relationships across teams and neighborhoods. The company seeks like-minded individuals who share their passion for building communities and making distributed computing accessible to everyone.

The new Deployment Graph API in Ray Serve enables fast local development to production deployment and is scalable with a unified DAG API across Ray libraries. It provides Python-native authoring experience, allows for easy composition of complex deployment graphs, and offers features such as independently scalable nodes, fractional resource allocation, shared memory, and dynamic dispatch. The API also supports function nodes, parallel calls, and optimization opportunities. With the Deployment Graph API, users can build, iterate, and deploy complex deployment graphs with ease, and Ray Serve will handle the serving aspect, including HTTP endpoint configuration and Python handle management. This new feature aims to simplify distributed computing and improve the overall developer experience.

Ray's runtime environments feature provides a simple way to manage files and packages on a cluster, making it seamless to scale up to a cluster while allowing for rapid iterative development. A runtime environment is specified in Python and can be easily described with an example, which includes the working directory, pip packages, environment variables, and more. This allows users to update their runtime environment along with their code updates without having to restart their Ray cluster or rebuild any container image. The feature also supports caching of files and packages on the cluster for quick reuse, making it efficient for concurrent workloads with different package dependencies. Additionally, features are planned to improve the functionality of runtime environments, such as better support for Docker images and cross-language support.

Siemens Technology has leveraged advancements in neural networks and reinforcement learning (RL) over the last two decades to create innovative products and services. The company's journey into RL began with a smart washing machine in 2003, and by 2017, it had launched proprietary software for controlling wind turbines and optimizing gas turbine performance. To bridge the gap between research and industry use of RL, Siemens employed simulations and digital twins, and opted for offline RL to minimize disruptions in real-world systems. The company prioritized transparency and human oversight in its RL solutions, ensuring trust and security in industrial settings. Through its efforts, Siemens has successfully applied RL to reduce delays and energy consumption in German metro trains.

JP Morgan AI Research is using reinforcement learning (RL) to model complex economic systems and efficient policy learning, which helps them simulate multiple heterogeneous interacting agents in a realistic market environment. The team needed an environment that could simulate the interactions of shops and consumers with diverse preferences, constraints, and connectivity. RL can help learn agent behaviors and policies and calibrate the agent composition with real-world data, addressing challenges such as finding equilibrium among strategic agents and model calibration. By using RL-based simulations built with Ray and RLlib, the team has greatly increased the efficacy and efficiency of their models.

Dow is doubling down on its digitization efforts, leveraging machine learning, advanced modeling techniques, robotics, and more to improve supply chain management and decision-making. Adam Kelloway's team at Dow's Digital Fulfillment Center is driving the development of reinforcement learning and mixed integer programming-based agents to enhance customer satisfaction, financial performance, and shareholder value. The AlphaDow project uses reinforcement learning-based agents for production scheduling on Azure compute clusters, overcoming challenges such as in-house design variables and deployment issues with the help of Ray libraries and capabilities. Through Adam's talk at the Production RL Summit, his team shares their experience and insights on how they turned AlphaDow into a viable solution that got the company onboard with the possibilities of reinforcement learning and machine learning.

At a recent Production RL Summit, Ben Kasper from Riot Games discussed how reinforcement learning (RL) has improved game balance in games such as Legends of Runeterra and DotA 2. By training an agent to play itself, Riot Games was able to identify issues that could ruin the game experience for players. The company employed a straightforward RL recipe that considered future variations in gameplay and captured patterns in game states. This led to the discovery of specific cards that were too strong or weak, allowing designers to balance the decks. The experiment showed promising results, with the RL-generated metrics matching design intuition and predicting the strongest deck. Riot Games has since productionized the process with an API, app, monitoring, and dashboard for analysis, making game designer KPIs include RL algorithm "balance" scores. This technology now enables designers to analyze and iterate on game balance prior to content release, and the company continues to update its tech stack to meet future challenges.