July 2024 Summaries

Meta's Llama 3.1 models, updated to support longer context lengths and new languages, benefit significantly from Unsloth's innovations that improve fine-tuning speed and reduce VRAM usage. Unsloth's enhancements make Llama 3.1 (8B) 2.1 times faster and 60% more memory efficient, while Llama 3.1 (70B) becomes 1.9 times faster with a 65% reduction in VRAM usage. Unsloth's approach allows fine-tuning on larger context lengths, which is particularly advantageous for Llama 3.1 (70B) when comparing its performance with other frameworks like Hugging Face and Flash Attention 2. The introduction of a new chat UI on Google Colab for Llama 3.1 models and the provision of pre-quantized 4-bit models facilitate faster downloads and improved interaction. Additionally, Llama 3.1's architecture now supports training other models using its outputs and employs fp8 precision for efficiency. These advancements underscore the importance of open-source development, as highlighted by Meta's commitment to enabling organizations to tailor models to specific needs without data exposure.

Unsloth has made significant updates to its offerings, particularly with the release of Mistral NeMo, a 12 billion parameter multilingual model that fits within a free Google Colab GPU instance for fine-tuning. The company has addressed several issues in NeMo, including the erroneous addition of an EOS token and dimension mismatches in the model's architecture, ensuring improved performance and VRAM efficiency. Additionally, Unsloth now supports CSV/Excel files with multicolumn datasets for fine-tuning, offers model deployment to Ollama, and has introduced a new documentation page. The platform has also enhanced long-context support across models such as Gemma 2 and Qwen2, and facilitates faster downloads from Hugging Face, alongside supporting Torch 2.5, Triton 3, and preparing for Flex Attention. These advancements underscore Unsloth's commitment to making fine-tuning more accessible and efficient.

Unsloth's latest advancements in finetuning Google's Gemma 2 models significantly boost performance and efficiency, allowing for faster processing and reduced VRAM usage compared to previous methods. The Gemma 2 (9B) model can now be finetuned twice as fast with 63% less memory, while the Gemma 2 (27B) achieves 1.9x faster finetuning with a 51% VRAM reduction. Unsloth also enables longer context lengths, up to 4-5 times for the 9B model, by implementing softcapping mechanisms that improve training accuracy and reduce VRAM usage. The integration of QLoRA and gradient checkpointing further enhances the training process, with updates to support Microsoft's Phi-3 mini update. Additionally, Unsloth has contributed fixes to the Gemma 2 Pytorch repository, addressing issues related to mixed precision training, and has actively participated in the AI Engineer World's Fair, engaging with the community through workshops and talks.