Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of Experiments

An experimental study of LoRA instruction fine-tuning used the Alpaca dataset, Llama 2 7B, and several language, factual, truthfulness, and arithmetic benchmarks to examine optimizer, quantization, training-duration, and adapter-hyperparameter choices on a single A100 GPU. QLoRA reduced peak GPU memory from about 21 GB to 14 GB but increased training time by roughly 30% and caused modest performance trade-offs, while replacing AdamW with SGD yielded little memory benefit at low LoRA ranks because relatively few adapter parameters were trainable. A cosine learning-rate scheduler improved some truthfulness and factual metrics, but training for two passes over the dataset generally degraded results, particularly arithmetic, likely because Alpaca contains limited arithmetic content. Expanding LoRA from attention components to all applicable layers and increasing rank improved results only when the alpha scaling parameter was adjusted proportionally; the strongest reported configuration used rank 256 and alpha 512, although very large ranks performed worse or failed to converge. The final leaderboard comparison found that tuned QLoRA improved several benchmarks over the base model but not all, underscoring that dataset selection, alignment methods, and evaluation diversity remain important alongside LoRA settings.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.