From Keywords to LLMs – The History of Search, and How ChatGPT Has Changed Search Forever

Search engines have evolved significantly from their early days of basic keyword matching to more sophisticated systems that incorporate natural language understanding and semantic search capabilities. Initially, search engines relied on keyword-based methods like Boolean matching and TF-IDF, which had limitations in ordering relevant documents and understanding user intent. To address these challenges, approaches such as Pseudo-Relevance Feedback and diversification techniques were developed, but these still faced issues like information drift. The advent of deep learning, particularly transformer-based models like BERT, revolutionized search by enabling the learning of query and document embeddings, allowing for more accurate semantic matches. However, the real paradigm shift occurred with the introduction of ChatGPT, which altered user expectations by providing concise, direct answers instead of lengthy lists of search results. Despite concerns about hallucinations in LLM-based systems, companies like Vectara are working on solutions like Grounded Generation to combine the strengths of traditional search and generative AI, offering hybrid searches that deliver relevant, context-aware answers with citations, thereby enhancing the user experience and overcoming language barriers.