November 2021 Summaries

The third generation of machine learning (ML) architectures has made significant progress in improving performance by exploiting the full bandwidth of distributed memory. Distributed libraries such as Ray Datasets and Ray Train enable improved performance by allowing for greater programmability, reducing operational and development overheads, and achieving better performance by passing data in-memory and with pipelining. A key capability of these architectures is their composability, which allows developers to compose existing distributed systems to solve complex problems. The example code snippet demonstrates how to create a Ray Dataset Pipeline with a distributed Ray Train Job, which can be used to express the ML ingest and training pipeline with just a few lines of Python code. This approach achieves lower operational and development overheads, better performance, and greater programmability compared to second-generation architectures.

The author shares their experience with great developer platforms that lead to extraordinary impact, citing Android as an example. They reflect on the challenges they faced in scaling machine learning problems at Uber AI and how it reminded them of the early days of the mobile ecosystem. The author then highlights Anyscale's vision to make distributed computing and scaling machine learning problems seamless and simple, using Ray to solve these problems. They express their enthusiasm for contributing to Anyscale's success and creating a product that delights users, while also highlighting the importance of the team and culture in achieving this goal.

The text discusses the importance of forecasting in business, particularly when lead times are long and inventory management is crucial. It highlights the challenges faced by data scientists when training models for time series forecasting, especially when dealing with large datasets that require frequent updates to account for changing data distributions. The authors propose using Ray, an open-source library developed at UC Berkeley, to parallelize and distribute Python code for ARIMA and Prophet models, making it easier to scale up forecasting tasks without rewriting the underlying codebase. Anyscale is also introduced as a tool for managing and running Ray applications in the cloud, providing a multi-cloud strategy with no vendor lock-in. The authors demonstrate how to use Ray and Anyscale to distribute ARIMA and Prophet training and inference tasks, achieving significant speedups in runtime.

Ray is an open-source project that allows users to parallelize Python processes and integrates with whylogs, a monitoring solution for machine learning models in production. Ray makes it easy to divide large datasets into smaller chunks and process them in parallel, generating whylogs profiles along the way. This enables users to monitor their ML models' performance and behavior in real-time. By using Ray pipelines and whylogs, users can easily integrate these tools into their workflows for data analysis and model monitoring.

A common grumble among data science or machine learning researchers is that putting a model in production is difficult, with 87% of models never seeing production due to lack of knowledge on how to deploy them effectively. To address this, it's essential to consider technical considerations and pitfalls when choosing an ML stack and tooling for model deployment. Using your laptop as a development environment can be beneficial, as long as the same code can run with minimal changes in a staging or production environment. Choosing a suitable programming language, such as Python, and an ML framework like PyTorch or TensorFlow, are also crucial considerations. Additionally, using feature stores to manage precomputed and cleansed features is vital for model accuracy. Model serving frameworks should be framework-agnostic, allow business logic, model replication, request batching, high concurrency, and low latency, with deployment CLI and APIs. Furthermore, models in production often operate in four ML patterns: pipeline, ensemble, business logic, and online learning, which require careful consideration when deploying them. Finally, model monitoring is critical to ensure a viable afterlife for the model, as data drifts, concept changes, failures, and system degradation can occur over time. Evaluating tools like Seldon, KFServing, Evidently.ai, Arize.ai, Arthur.ai, Fiddler.ai, Valohai.com, or whylabs.ai to meet these considerations is essential for successful model deployment.

With the explosive growth of data, C++ is widely used in modern distributed systems due to its high performance and lightweight characteristics. Building a C++ distributed system with complete features and high availability for production environments requires addressing issues such as communication, deployment, and fault tolerance. Ray's C++ API was designed to help address these issues, providing a low-level and flexible API that translates existing concepts of functions and classes into tasks and actors. The API allows users to focus on the logic of the system itself. Ray is an open-source library for parallel and distributed Python, but it also supports C++. Its Java API was contributed by Ant Group in 2018, and its C++ API was added recently. The C++ API can be used to seamlessly connect the user layer and the core layer, resulting in a system with no inter-language overhead. Ray Tasks are stateless, and actors can be used for stateful computing. Ray provides advanced features such as scheduling options, fault tolerance, deployment, and operation, and its dashboard allows users to view the state of the Ray cluster and applications in real time. The project template for C++ applications is available, which includes a wheel packaged with pip and a bazel-based template.

The cidaas ID validator uses a combination of artificial intelligence and machine learning models to verify digital identities, ensuring that users are who they claim to be by analyzing biometric data, security features, and identity documents. The system is fully automated, requiring users to record videos of their face and identity documents in a guided process. This process involves multiple steps, including liveness detection, real-time checks, and verification of the user's identity against their ID document. The cidaas ID validator complies with European standards for digital identification and has been successfully implemented using Ray Serve, which provided scalability, reduced framework usage, and lower maintenance efforts, ultimately increasing the processing speed by a factor of four.

Ray version 1.8 has been released! The distributed deep learning library Ray SGD has been rebranded to Ray Train, allowing easy scaling of model training using Pytorch, Tensorflow, or Horovod. A tight integration with Ray Datasets for distributed data loading while training is also a new feature, supporting use cases such as handling large datasets and minimizing cross-node data transfer. Additionally, experimental support has been added for Ray on Apple Silicon (M1 Macs), enabling optimal performance on these chips. The release includes various other features and enhancements, which can be learned about by visiting the release notes.

The authors from Intel's AI and Analytics Team introduce BigDL, an open-source framework for building scalable end-to-end AI applications on distributed big data using Ray. BigDL seamlessly integrates Ray into big data preprocessing pipelines through RayOnSpark, allowing users to run AI applications directly on their existing big data clusters in a production environment. The authors also showcase the AutoML capability built on top of Ray Tune, which enables users to automate hyperparameter tuning and model selection for various AI applications. Additionally, they introduce Project Chronos, a framework for building large-scale time series analysis applications using BigDL, RayOnSpark, and AutoML capabilities.