Fine-tuning Gemma 3 on a Custom Web Dataset With Firecrawl and Unsloth AI

Gemma 3, Google's latest advancement in open-weight models, represents a significant leap in the evolution of language models, particularly with its release in March 2025. This ambitious model family comes in four sizes, with both pre-trained and instruction-tuned variants, and introduces true multimodality by handling images and text, a first for the Gemma series. Notably, the Gemma-3-27B-IT model surpasses previous versions and competitors like Gemini 1.5-Pro in benchmark tests while using fewer computational resources. Key technical enhancements include context windows up to 128K tokens, support for over 140 languages, and advanced vision capabilities powered by Google's SigLIP encoder. The tutorial explores using Firecrawl for dataset creation and Unsloth for efficient fine-tuning of Gemma 3, emphasizing the importance of high-quality, diverse datasets for successful language model fine-tuning. The process involves transforming datasets into instruction-response formats, using tools like huggingface_hub and transformers for model handling, and leveraging libraries such as unsloth and peft for optimized training on consumer hardware. Despite its sophisticated capabilities, the tutorial highlights challenges in model output quality, underscoring the critical role of data quality and parameter tuning in achieving high-performing fine-tuned models.