June 2023 Summaries

The RedPajama project has released its v1 versions of models, including instruct-tuned and chat versions, under the Apache 2.0 license. The RedPajama-INCITE-7B-Instruct model outperforms all open 7B models on HELM benchmarks by 2-9 points, making it ideal for a wide range of tasks. The chat model is built on fully open-source data and does not use distilled data from closed models like OpenAI's, ensuring it is clean for use in open or commercial applications. A base model was also trained on the RedPajama dataset, with the same architecture as the popular Pythia model suite, but slightly behind Falcon-7B on HELM. The project aims to make future open-source models even better through community feedback and improvements on the data side, including balancing data mixture over each data slice and exploring further data deduplication strategies.

Fine-tuning language models over slow networks using activation compression with guarantees` AC-SGD, a novel activation compression algorithm for communication-efficient pipeline parallelism training over slow networks, compresses the changes of activations instead of values, achieving O(1/T‾‾√) convergence rate without assuming gradient unbiasedness. AC-SGD can be optimized and implemented efficiently, providing up to 4.3X end-to-end speed-up in slower networks without sacrificing model quality. When combined with state-of-the-art gradient compression algorithms, AC-SGD enables "end-to-end communication compression" for significant speed-ups, with up to 4.9X improvement. This technique offers a cost-effective approach (20% faster training) and can be applied to large-scale models (up to 1.5 billion parameters), making it suitable for various applications, including those requiring high-quality datasets like RedPajama-V2.

This paper presents a novel approach for training large foundation models in decentralized heterogeneous environments, where different computational "tasklets" are allocated to devices connected by slow networks. The authors propose a scheduling algorithm and formal cost model to optimize the allocation strategy, achieving significant speedup over prior state-of-the-art systems. Extensive experiments demonstrate that their approach can reduce training time by up to 4.8X compared to existing methods, while also providing efficient network compression. By leveraging decentralized and heterogeneous networks, this work aims to make large-scale foundation model training more accessible and cost-effective.