Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes - Summary

The paper introduces a novel mechanism called "Distilling step-by-step" that enables smaller models to outperform larger language models (LLMs) by utilizing less training data and reduced model sizes. This mechanism leverages LLM rationales as additional supervision within a multi-task training framework, leading to improved performance across four NLP benchmarks. Notably, the approach surpasses both fine-tuning and traditional distillation methods by requiring fewer labeled and unlabeled training examples. The findings highlight that smaller models not only outperform LLMs in terms of efficiency but also do so with significantly smaller model sizes and reduced data requirements.