December 2025 Summaries

Moonshot AI has introduced Kimi Delta Attention (KDA) as an extension of Gated DeltaNet, offering promising results particularly in local and global attention hybrid arrangements. The author experiments with converting the AFM-4.5B-Base model into a hybrid KDA and full-attention transformer using knowledge distillation, inspired by the RADLADS paper. The process involves creating a student model, modifying the attention architecture, and using a streamlined distillation pipeline to achieve efficient alignment with the teacher model. The experiments highlight the advantages of KDA in certain contexts, particularly long sequence lengths, while also noting performance variations based on attention configurations. The introduction of the open-source DistillKit facilitates these experiments, enabling both online and offline distillation with various loss functions and model configurations, demonstrating the potential for faster and more efficient model training and inference.

Arcee AI has launched Trinity, an open-weight mixture-of-experts (MoE) model family, aiming to establish a fully U.S.-trained AI framework that businesses and developers can own. The initial models, Trinity Nano and Mini, are available now, with the much larger Trinity Large expected in January 2026, trained on 2048 B300 GPUs. While Trinity Mini is a fully post-trained reasoning model, Trinity Nano Preview is an experimental chat model optimized for sparsity and small deployments, albeit potentially unstable in edge cases. Trinity's development underscores the importance of owning the pretraining process to ensure legal compliance, improve product longevity, and facilitate AI systems that self-improve through direct user interaction. The models leverage advanced architectural techniques such as gated attention and are trained on a carefully curated dataset in partnership with DatologyAI and Prime Intellect. As Arcee AI progresses toward releasing the 420B parameter Trinity Large, the company invites the community to test and provide feedback on Nano and Mini to refine future models.