January 2025 Summaries

DeepSeek-R1, an open-source model rivaling OpenAI's O1 reasoning model, has been optimized for local use through a process called dynamic quantization, reducing the model size from 720GB to 131GB while maintaining functionality. This optimization involves selectively quantizing certain layers to higher bits and leaving most mixture of experts (MoE) layers at 1.5 bits, which helps prevent performance issues like endless loops and incorrect outputs. Users can run the model without a GPU, although it will be slow, with optimal performance requiring a combination of VRAM and RAM totaling at least 80GB. Benchmarks show that the 1.58-bit version achieves a 69.2% score on a Flappy Bird game benchmark, compared to 91.7% for a 2-bit version, while naive quantization approaches lead to poor performance. The DeepSeek-R1 architecture exploits MoE layers to increase parameters without raising computational costs, and specific layers are left at higher precision to maintain accuracy. The model and its components are available on platforms like Hugging Face, and detailed instructions for running and optimizing the model are provided for different hardware configurations.

DeepSeek's new R1 model sets impressive benchmarks in reasoning performance, equaling OpenAI's o1 model and building on the previously launched DeepSeek-V3. The model has been distilled and fine-tuned on Llama 3 and Qwen 2.5, allowing users to fine-tune it with Unsloth. The R1 series includes various versions like GGUF's and a new 1.58-bit Dynamic GGUF that reduces size by 80%. Running DeepSeek-R1 requires llama.cpp, and while a GPU isn't necessary, a CPU with substantial RAM and disk space is essential. The DeepSeek team has made smaller, distilled models available for local use, and fine-tuning options are provided using compatible Llama and Qwen architectures. They also offer resources like Colab notebooks for free fine-tuning, encouraging community engagement through platforms like Reddit and Discord.

Microsoft's new Phi-4 model, integrated into Unsloth, rivals OpenAI's GPT-4o-mini in performance and has recently undergone significant improvements to enhance its accuracy. These enhancements include resolving four key bugs related to tokenization, fine-tuning, and chat template issues, while also converting the model to Llama's architecture for improved accuracy and ease of use. The fine-tuning process is now twice as fast, uses 70% less memory, and supports context lengths over 128K, which is substantially longer than previous models. The updates also introduce dynamic 4-bit quantization, which boosts accuracy without a significant increase in VRAM usage. These changes have been well-received, with feedback from Reddit users indicating improved model performance on the Hugging Face OpenLLM Leaderboard. Users can experiment with fine-tuning Phi-4 using Unsloth's Colab Notebook, which is compatible with Google's free Tesla T4 16GB GPU, providing an accessible platform for further customization and testing.