October 2021 Summaries

Hutom is a big data platform that helps hospitals and surgeons provide more optimized patient care by leveraging machine learning and computer vision. The platform's automated system assists in better personalized surgical planning, real-time surgeon assistance, surgery analysis and review for learning and archival purposes. However, training deep learning models on medical data poses several technical challenges such as acquiring medical data due to legal or administrative requirements, high cost of annotation, small non-diverse datasets, and performance gains from scaling up computing resources. To overcome these challenges, Hutom utilized a synthetic data generation technique, implemented domain randomization and semi-supervised methods, and leveraged Ray and its ecosystem for distributed training and hyperparameter search. The platform's use of Ray Tune, which is a hyperparameter tuning library built on Ray, enables easy launch of multi-node distributed hyperparameter sweeps with features such as automatic management of checkpoints and logging to TensorBoard. Additionally, the platform uses Ray Train, a lightweight library for distributed training of deep learning models, to enable distributed training on GPUs. After obtaining the best performing model from hyperparameter search, the model is deployed using Ray Serve, which enables easy scaling of surgical video analysis, review, and deployment. The deployment utilizes Ray Serve's batching feature to perform computation in parallel, resulting in a 29% reduction in latency by increasing batch size.

Ray version 1.7 has been released with several key updates, including the introduction of alpha versions for Ray SGD v2 and Ray Workflows, which aim to improve usability, composability, and performance for large-scale data science, ML, and long-running business workflows. The new SGD v2 offers a simpler API, better integration with other libraries, and more features, while the Workflows library provides flexibility, high-performance execution, durability, and dependency management. Additionally, a C++ API has been introduced, enabling users to build C++ distributed systems easily and reducing inter-language overhead. These updates aim to provide a more streamlined experience for users and support various use cases with Ray's core capabilities.

Ray Serve is a web framework specialized for ML model serving that aspires to be easy to use, easy to deploy, and production ready. It provides scalability, multi-model composition, batching, FastAPI integration, and framework-agnostic support. Ray Serve helps with the tradeoff between ease of development and production readiness in the ML serving space by providing a simple and elegant API for deploying and managing ML models. It natively supports online learning, ensemble patterns, business logic patterns, and authentication and input validation. With Ray Serve, you can compose multiple models together, scale out each component individually, and load balance calls across replicas, making it easier to leverage Ray for complex architectures involving many models spanning multiple nodes.