To Reason or Not to Reason: Is 5% more accuracy worth >5x cost?

The experiments explored the impact of training large language models (LLMs) with reasoning data on tasks like data transformation and information extraction, focusing on performance improvements and associated costs. It was found that fine-tuning models with reasoning traces could enhance output quality, but this benefit was primarily observed in models already trained with reasoning capabilities. Conversely, finetuning models lacking such prior training could degrade performance. Additionally, models trained with reasoning traces generated significantly more tokens, leading to increased computational costs and latency. Techniques like Chain-of-Thought prompting, inference-time scaling, and reinforcement learning were highlighted as methods to improve LLM reasoning capabilities. The study underscored the need to balance performance gains with the increased cost and latency, as the average improvement in output quality was 4.9%, accompanied by a substantial increase in token generation and associated costs.