Context Rot Is Real: Why Your AI Gets Worse Over Time

Context rot is a phenomenon that occurs when AI systems, particularly those integrating large language models (LLMs), degrade over time due to the accumulation of outdated or conflicting information without appropriate updating or filtering. This degradation manifests subtly, often leading to longer prompts, increased latency, and technically plausible yet incorrect answers. The common remedy of adding more context information exacerbates the issue by overwhelming LLMs with irrelevant data, highlighting the inefficiencies of vector-based retrieval methods. In contrast, graph-based retrieval systems, like GraphRAG, offer a more structured approach by maintaining coherent and current context through connected entities, which helps reduce confusion and improve the accuracy of AI outputs. The challenge of context rot extends to the tools and protocols used by AI systems, where evolving permissions and business logic further complicate data retrieval and processing. Addressing context rot requires disciplined context engineering, where information is carefully curated, updated, and prioritized to ensure the reliability and relevance of AI system outputs over time.