November 2022 Summaries

At Together, researchers are working to bring the world's computation together in a decentralized cloud to enable AI research and improvement. Decentralized training of foundation models poses significant challenges due to high network bandwidth requirements. To address these issues, two papers were presented at NeurIPS 2022 focusing on optimizing efficiency for decentralized training over slow networks. The first paper addresses scheduling in heterogeneous environments, while the second focuses on communication compression. By optimizing the efficiency of decentralized training, researchers aim to significantly reduce costs associated with training foundation models. In an empirical study, the authors found that even with a network 100x slower than data center networks, the end-to-end training throughput is only 1.7-2.3x slower for GPT-style models with 1.3B parameters. The results demonstrate the potential of scheduling and system optimizations to bridge the gap between decentralized and data center training. However, beyond scheduling, other aspects such as fault tolerance, network jitters, heterogeneity on devices, and communication compression require attention for end-to-end systems.

The GPT-JT model was developed by Together as a decentralized, open-source AI fork of GPT-J-6B, fine-tuned on 3.53 billion tokens to outperform most 100B+ parameter models at classification tasks. It was trained with a new decentralized algorithm using slow internet connections and heterogeneous GPU hardware, resulting in improved performance and reduced communication overhead compared to traditional distributed learning algorithms. The model's strength lies in its ability to leverage community projects and datasets, such as EleutherAI's open models and Google Research's UL2 technique, to achieve state-of-the-art results. GPT-JT is now publicly available as open source, along with a live demo on the HuggingFace space, and offers a value chain that everyone can benefit from. The model's decentralized training approach paves the way for making AI more accessible to researchers and practitioners via decentralized computing, reducing costs and increasing efficiency.

The Stanford Center for Research on Foundation Models (CRFM) has announced a comprehensive effort to benchmark 30 language models, known as Holistic Evaluation of Language Models (HELM), which covers 42 scenarios from question answering to sentiment analysis. To support this effort, the Together Research Computer aggregates idle GPU cycles across thousands of servers, running inference over more than 11 billion input tokens and 1.6 billion output tokens on 10 open language models, including GPT-3, Flan-T5, BLOOMZ, and Galatica. This decentralized computing approach aims to make the field more accessible to researchers and practitioners by solving computation bottlenecks in large language models. The project is an early step towards enabling efficient, shared compute for AI, with the goal of bringing the world's compute together to enable everyone to contribute to and benefit from advanced AI models.