Context pruning: cut LLM tokens without losing quality

Context pruning is a technique in large language models (LLMs) that involves selectively removing low-value tokens, sentences, or passages from input data to reduce costs and improve response quality. This method falls under the broader category of prompt compression and is distinct from prompt engineering, model pruning, and abstractive summarization, as it focuses on modifying the input rather than the model or generating new text. Various pruning strategies include token-level, sentence-level, attention-based, and dynamic layer-progressive pruning, each with unique advantages and trade-offs. While larger context windows do not inherently resolve issues associated with lengthy inputs, effective context pruning can lead to significant improvements in processing efficiency and output quality. Additionally, the integration of semantic caching can further optimize the process by utilizing past query responses to minimize redundant computation. However, context pruning requires careful implementation to avoid potential drawbacks like information loss and increased hallucination, especially when handling structured data, multi-turn conversations, and compounded optimization scenarios.