The Underdog Revolution: How Smaller Language Models Can Outperform LLMs

Recent research suggests that smaller language models (SLMs) are starting to outperform or match the performance of large language models (LLMs) in various applications, despite their larger counterparts' remarkable natural language understanding and generation capabilities. SLMs have several advantages over LLMs, including faster training and inference speeds, lower energy consumption, and reduced memory requirements. These efficiency benefits extend to other aspects of SLM use, such as smaller carbon and water footprints. As the focus shifts towards making AI more accessible and compatible with a broad range of devices, SLMs are becoming increasingly important in shaping the future of AI. Techniques like transfer learning, knowledge distillation, and specialized masking techniques have been employed to enhance the performance of SLMs. The potential for smaller models to achieve impressive performance gains without large-scale investment is showcased by recent techniques proposed by Google, UL2R, and Flan.