The RedPajama project aims to create a set of leading open-source models by rigorously understanding the ingredients that yield good performance. The project has released the RedPajama-INCITE family of models, including base, instruction-tuned, and chat models. The 3B model is the strongest in its class, with the small size making it extremely fast and accessible. The instruction-tuned versions achieve strong performance on HELM benchmarks. The 7B model outperforms the Pythia 7B model, demonstrating the value of a bigger dataset. The project plans to build models at larger scale using the new dataset, which will go beyond the quality of LLaMA 7B. These models are released under the Apache 2.0 license, allowing for use in both research and commercial applications.