July 2023 Summaries

Made With ML, a platform focused on helping machine learning teams deliver value with ML in production, is now part of the Anyscale team. Anyscale is behind the popular framework Ray, which helps scale and productionize AI applications, especially large language models (LLMs) and generative AI. The new partnership aims to make it easier for developers to develop and productionize ML workflows, addressing challenges such as scaling, integrations, and dev-to-prod pipelines. Made With ML's platform will continue to provide a community-driven space for learning and improving ML practices, with the goal of accelerating the maturity of ML workflows and promoting responsible ML development. The Anyscale team has been working on these issues with top ML teams worldwide, and this partnership enables further progress in addressing obstacles such as data size, model size, serving volume, integrations, and dev-to-prod handoffs.

The latest release of Ray 2.6 introduces several key enhancements across the ecosystem, focusing on improving real-time capabilities, distributed training, and persistence. Streaming responses in Ray Serve enable applications to return results incrementally, enhancing user experience for computationally expensive workloads like large language models. Batch requests are also supported, allowing users to utilize hardware resources more efficiently. Additionally, Ray Data's streaming lazily execution is integrated with Ray Train, reducing memory usage during training. The release also introduces a new multi-GPU Learner API in the PPO algorithm, providing a simpler and more powerful alternative. Furthermore, improvements have been made to ensure reliability and persistence of training artifacts, including support for cloud storage and NFS paths. Overall, these changes aim to enhance performance, stability, and ease of use for users, while reducing network latency and memory usage.

The Anyscale Snowflake Connector enables easy data transfer between Snowflake clusters and Anyscale-hosted Ray clusters, facilitating machine learning discovery and development within Anyscale while leveraging existing Snowflake data lake capabilities. This connector simplifies the creation of end-to-end machine learning workloads by allowing the entire ML pipeline to be executed within a single Python script. It also enables faster execution of machine learning workloads such as training, tuning, and batch serving jobs due to Ray's highly scalable nature. The connector provides simple access to Snowflake data through SQL queries in Anyscale Workspaces Visual Code or JupyterHub environments, while ensuring data security and governance by directly copying data from Snowflake to Ray clusters. By leveraging the parallel read and write capabilities of Ray datasets, large datasets can be queried and transferred quickly into a Ray dataset distributed across the Ray cluster. This enables Data Scientists and Machine Learning Engineers to develop unified workloads with a single Python script that scales across a Ray cluster, integrating with popular AI and ML libraries such as Hugging Face, XGBoost, LightGBM, PyTorch, and TensorFlow.

Fine-tuning is one approach to domain-specific model refinement (DSMR), but it's not a silver bullet for improving domain-specific performance. When customers ask about fine-tuning, they're often describing a job that involves improving the quality of the model's output based on their specific application needs. However, fine-tuning typically results in creating a niche model for a niche use-case, and it's most effective when the data is plentiful and easy to come by. The approach doesn't work well when it comes to facts and hallucination. Instead, there are broader DSMR techniques available, including prompt refinement, example selection, retrieval assisted generation, and reinforcement learning with human feedback. These approaches can help solve problems that require incorporating new concepts into the internal knowledge graph represented inside the neural network.