April 2024 Summaries

Unsloth has introduced updates to the Llama 3 model, significantly enhancing its performance and efficiency in fine-tuning processes. The Llama 3 (8B) model can now be fine-tuned twice as fast while using 63% less memory compared to previous solutions like Flash Attention 2 (FA2) and Hugging Face, and Llama 3 (70B) achieves 1.8 times faster processing with a 68% reduction in VRAM. Notably, these models now support much longer context lengths, with the Llama-3 70B model fitting 48,000 tokens compared to 7,000 without Unsloth. They have made a Colab notebook available for finetuning the 8B model using a free Tesla T4 GPU, and pre-quantized models are available for faster downloads. Additionally, the update addresses quirks such as the tokenizer not adding the BOS token and dealing with untrained tokens in the base model to prevent issues during finetuning. The community is encouraged to support and engage with Unsloth through various platforms, including Discord and Twitter, while also looking forward to future updates like Phi 3 support.

Unsloth Gradient Checkpointing is a new algorithm designed to enhance the fine-tuning of large language models (LLMs) by enabling exceptionally long context windows, achieving up to 228,199 tokens on NVIDIA H100 80GB GPUs and 56,420 tokens on RTX 4090 24GB GPUs. This innovative approach reduces memory usage by 30% with a minimal time overhead, making it significantly more efficient for fine-tuning LLMs with long contexts compared to previous methods. The algorithm is compatible with all model architectures utilizing gradient checkpointing, and it allows for a substantial increase in batch size. Additionally, improvements in tokenizer efficiency and RoPE Embeddings speed contribute to enhanced performance, while the introduction of pre-quantized models further reduces download time and VRAM usage. The developers also highlight upcoming features such as an automatic model optimizer and a Colab-enhanced one-click fine-tuning system, and they encourage community engagement and support through platforms like Discord and Ko-fi.