Did filesystem tools kill vector search?

Vercel's recent experiment demonstrated that reducing the number of tools in a text-to-SQL agent by leveraging filesystem tools like cat, ls, or grep can significantly simplify processes and enhance performance. They discovered that filesystems inherently address many issues that complex tools aim to solve, prompting a shift towards letting large language models (LLMs) navigate filesystems autonomously. LlamaIndex also explored this approach, comparing a filesystem-exploring agent with a traditional Retrieval Augmented Generation (RAG) system using a benchmark dataset of AI/ML papers. The fs-explorer agent, which uses tools like read_file, grep_file_content, and LlamaParse, excelled in correctness and relevance due to its access to full file content but lagged in speed compared to RAG, which benefits from optimized, fast network calls. When scaled to handle larger datasets, RAG maintained an edge in speed and correctness, suggesting that while agentic file search offers more context, it may not be suitable for real-time applications where latency is critical. The decision between these approaches depends on the specific requirements of the task, as RAG can handle greater complexity and scale, whereas filesystem-based agents deliver high-quality results by leveraging extensive context.