August 2023 Summaries

This startup, ThirdAI Corp, has developed a new deep learning framework called BOLT, which efficiently trains large models on standard CPU hardware by making sparsity a first-class design principle. The company leveraged Ray for distributed training of their models, achieving near-linear scaling for terabyte-scale datasets and billion-parameter models. By using Ray's distributed data parallel engine, ThirdAI was able to quickly build an industry-grade solution with features like fault-tolerance, multiple modes of communication, and seamless scalability. This approach allows for the democratization of deep learning in a sustainable manner, as specialized hardware is not required, reducing costs and energy consumption. The company has also simplified their developer experience by transitioning from Ray Core to Ray Trainer, which provides a streamlined training pipeline, enhanced fault tolerance, and refined automatic scaling. Experimental results demonstrate the performance of BOLT on various benchmarks, showcasing its competitive efficiency on CPUs.

Anyscale Endpoints has made experimentation with LLMs more accessible, allowing researchers to compare the factual accuracy of different models, including open-source LLMs like Llama 2. The comparison showed that Llama-2-70b is almost as strong as gpt-4 in terms of factuality and considerably better than gpt-3.5-turbo. However, Llama 2-7b and Llama 2-13b had severe ordering bias issues, while gpt-3.5-turbo showed a significant ordering bias. The cost comparison revealed that Llama 2 is 30 times cheaper for summarization than gpt-4, despite having similar performance levels. This experiment highlights the importance of considering the ordering bias when using LLMs for summaries and the potential benefits of using open-source LLMs like Llama 2.

We leverage multi-modal models to enable applications such as text-based image retrieval or object detection, powering various use cases at ByteDance, including large model offline inference. To handle the scale of our workload, we utilize Ray as a computing framework, specifically Ray Data, which provides flexibility and scalability for large-scale model parallel inference. We employ pipeline sharding, splitting our model across GPU devices to fit within memory constraints. This approach allows us to overcome technical challenges posed by data size and model size. By utilizing Ray Data's streaming execution paradigm and elastic resource scheduling, we can achieve high efficiency in building scalable offline inference applications for large models. Additionally, we leverage KubeRay to manage our Ray clusters, providing a comprehensive solution for deploying and managing Ray applications.

The fine-tuned models consistently outperform the non-fine-tuned base models across all tasks, demonstrating that fine-tuning can significantly enhance performance for specific tasks. Fine-tuned models also have the potential to be more cost-effective in the long run compared to using general-purpose models like GPT-4 or Llama-2 chat models, as they may require fewer tokens and thus lower costs during serving.

Anyscale is releasing a preview of Anyscale Endpoints, a serverless approach to running open-source Large Language Models (LLMs) fast, cost-efficiently, and at scale. This allows developers to integrate LLM APIs with ease, streamlining the deployment and fine-tuning process. The platform offers state-of-the-art performance and cost optimizations, seamless integration with cloud services, and a comprehensive suite of services for secure and customized environments. With Anyscale Endpoints, developers can accelerate their LLM application development, focusing on innovation while leaving infrastructure management to the platform.