Company
Date Published
Author
-
Word count
1563
Language
-
Hacker News points
None

Summary

In 2019, Elasticsearch introduced the WAND algorithm to enhance the speed of retrieving top hits in search queries, building on earlier work with the MAXSCORE algorithm. Originally presented by Stefan Pohl, MAXSCORE aimed to optimize search by skipping non-competitive documents, but its integration was hindered by the need for static indexes. The implementation of WAND, which uses a more refined approach by assigning weights to query clauses, allowed for dynamic index optimization, significantly speeding up query execution. Further improvements were made by adopting block-max WAND, a variant that handles outlier scores more efficiently by splitting postings into fixed-size blocks and recording maximum impact scores per block. This development led to remarkable speedups in various query types, though it introduced changes such as potentially less accurate total hit counts and the requirement for non-negative scores. These optimizations are set to be incorporated into Lucene 8.0 and Elasticsearch 7.0, offering substantial performance benefits while allowing users to adjust the number of hits counted to balance speed and accuracy.