Company
Date Published
Author
Vivek Sourabh
Word count
1069
Language
English
Hacker News points
None

Summary

The history of search has evolved significantly over time, from basic keyword matching to large language models (LLMs), which have revolutionized the way users interact with search systems. The earliest search engines relied on Boolean Matching and TF-IDF, but these methods had limitations in producing an ideal order of relevant documents. Researchers developed approaches such as importance of words, pseudo-relevance feedback, and diversification to address these issues. However, it wasn't until the introduction of transformer-based models like BERT that natural language understanding capabilities were integrated into search systems. The development of Approximate Nearest Neighbor search libraries like FAISS helped improve the performance of LLM-based search systems. With the advent of ChatGPT, users now expect single-line/paragraph answers, which has led to questions about the obsolescence of traditional search systems. However, researchers are working on addressing hallucinations and developing "Grounded Generation" techniques that combine the strengths of both keyword-based and LLM-based searches.