April 2023 Summaries

CocktailSGD is a novel communication-efficient training framework designed to train large language models (LLMs) over slow networks, such as 500Mbps connections. This approach combines three distinct compression techniques - random sparsification, top-K sparsification, and quantization - to achieve much greater compression than individual techniques alone. Theoretical analysis justifies the benefit of this hybrid approach, while empirical results show that CocktailSGD achieves up to 117x compression in fine-tuning LLMs without compromising convergence. On a slow network, CocktailSGD only incurs a small slowdown compared to data center networks. The RedPajama-V2 Dataset is conceptualized as a foundation for creating high-quality datasets, and its use requires filtering out data using quality signals that accompany it, depending on the intended application.

We have made significant progress in training our RedPajama model, which aims to create leading open-source models. We trained a 1T base dataset, independent of the model architecture, and compared it to the Pile dataset used for open pre-training data. Our results show that the RedPajama dataset outperforms the Pile dataset with respect to certain benchmarks, particularly at higher token counts. The quality of our model checkpoints continues to improve as we train on more tokens, but we still lag behind LLaMA-7B in some metrics. We are excited about continuing to improve the data and exploring ways to combine it with other datasets. Our goal is to work with the open-source AI community to build the best large language models possible.

Together and Crusoe are partnering to build a state-of-the-art generative AI platform using carbon-reducing compute resources, aiming to minimize the environmental impact of AI model training. This initiative involves building an open ecosystem that promotes sustainability and reduces the carbon footprint of AI models. The Together Decentralized Cloud is being developed with energy-efficient nodes distributed across the Internet, while Crusoe's Digital Flare Mitigation solution harnesses flared natural gas to power data centers, reducing methane emissions and energy waste. This partnership enables the creation of a green zone within the cloud, where compute resources significantly reduce carbon emissions, supporting the development of high-quality AI models for humanity.

RedPajama` is a collaborative project aiming to create leading, fully open-source models, following in the footsteps of `Stable Diffusion`, which demonstrated the potential of open-source models. The project starts by reproducing the `LLaMA` training dataset of over 1.2 trillion tokens, with the goal of creating a set of high-quality, fully open-source models that can rival commercial offerings. The first component released is the pre-training data, which has been carefully filtered and processed to ensure its quality and broad coverage. The project involves multiple collaborations between organizations and researchers, including `Together`, `Ontocord.ai`, `ETH DS3Lab`, `Stanford CRFM`, and `Hazy Research`. The dataset consists of seven data slices, each with its own filtering process, and is available for download through `Hugging Face`. With the pre-training data released, the next step is to train a strong base model and instruction tune it using various tools and techniques. The project acknowledges the contributions of the open-source AI community and recognizes the potential of fully open-source models in removing limitations on research, customization, and sensitive data use.