Home / Companies / Unsloth / Blog / Post Details
Content Deep Dive

Fixing All Gemma Bugs

Blog post from Unsloth

Post Details
Company
Date Published
Author
Daniel & Michael
Word Count
745
Language
English
Hacker News Points
-
Summary

Unsloth has dedicated the past week to addressing and resolving various bugs in the Gemma model, initially highlighted by Google. These bugs included issues with loss value discrepancies, precision problems with embedding calculations, and improper casting in Keras, particularly affecting the RoPE and GELU functions. Efforts have been made to correct these through updates to Colab notebooks and ongoing pull requests, such as the approximate GELU and RoPE fixes, which are now partially resolved in version 4.38.2 of the Hugging Face transformers. Despite being a small team without external funding, Unsloth continues to share progress and invites support through donations and community engagement on platforms like Discord and Twitter.