Auto-Evaluation of Anthropic 100k Context Window

Lance Martin discusses the potential of retriever-less architectures in LLM question answering (Q+A) due to the emergence of models with significantly larger context windows, such as Anthropic's 100k token model, which can handle entire documents without a separate retrieval step. This development raises questions about the necessity of traditional retrieval methods, especially for smaller document sets. Testing reveals that while the Anthropic 100k model performs well against retrieval-based methods like kNN and SVM in some cases, it suffers from higher latency and occasionally less accurate answers due to the inability to inspect retrieved chunks. Despite these challenges, retriever-less architectures offer simplicity and are promising for applications with manageable document sizes and where latency is less critical, as LLM context windows continue to expand and models become more efficient.