December 2023 Summaries

Unsloth has announced the release of QLoRA support for models based on the Llama architecture, enhancing training speed and reducing memory usage. The open-source release includes models like Mistral 7B and CodeLlama 34B, with significant speed improvements noted on GPUs such as the A100 and T4. The Unsloth Pro version offers further enhancements, enabling faster fine-tuning and greater VRAM savings. Innovations such as sliding window attention and preliminary Windows and DPO support have been integrated, with a focus on optimizing various computational aspects, including reducing data upcasting and using more efficient implementations of attention mechanisms and embeddings. The platform also provides extensive benchmarking comparisons and shared notebooks for reproducibility, alongside ongoing developments for additional features and support.