December 2023 Summaries

SuperAGI has developed SAM (Small Agentic Model), a 7B model that exhibits strong reasoning capabilities, outperforming larger state-of-the-art models like GPT-3.5 on multiple reasoning benchmarks such as GSM8k and ARC-C. Despite being trained on a significantly smaller dataset, SAM-7B achieves superior performance by leveraging a strategically curated dataset and explanation traces to enhance its reasoning abilities. The model was fine-tuned using open-source large language models (LLMs) and evaluated with a custom pipeline to ensure fair comparison across benchmarks like HellaSwag and Winogrande. The research emphasizes the importance of data quality over quantity and introduces a method of generating reasoning traces through a Reasoning Skeleton Prompt to guide open-source LLMs in producing consistent and diverse explanations. Although SAM-7B excels in reasoning tasks, it is not suited for conversational applications due to the absence of safety and bias guardrails, indicating that while the model is promising for reasoning, it requires further development for broader applications.

Positional Skip-wisE (PoSE) training is introduced as an efficient method to extend the context window of Large Language Models (LLMs) without the high computational costs associated with full-length fine-tuning. Unlike traditional methods such as Position Interpolation, PoSE manipulates position indices to simulate longer inputs within a fixed context window, minimizing memory and time overhead while maintaining performance. This approach was successfully applied to extend the context window of the Mistral 7B model from 8K to 32K, demonstrating its effectiveness in language modeling and information extraction tasks with minimal performance degradation. PoSE is compatible with all RoPE-based LLMs and position interpolation strategies, providing a cost-effective solution for handling extremely long contexts. The model employing PoSE is available on Hugging Face, validating its practical application and empirical success.