Search Behind-the-Scenes: How Neeva Uses Human Evaluation to Measure Search Quality

Neeva, a private and ad-free search engine founded by former Google employees, emphasizes the importance of search quality measurement over user engagement metrics like clickthrough rate, which can inadvertently lead to irrelevant, clickbait content. Instead, Neeva employs human raters to evaluate the relevance of search results, particularly for technical programming queries, by comparing its performance to Google's. This evaluation involves a personalized approach, using a programming-specific rating team and tailored query sets from raters' personal browsing histories. Neeva's focus on rich user interfaces, such as providing direct code snippets in search results, often distinguishes it from Google, although Google's superior handling of long-tail queries is noted. The insights gained from these evaluations contribute to improving search algorithms, enabling offline experimentation, and forming high-quality datasets for machine learning training, ultimately aiming to optimize search engines for user satisfaction and relevance rather than mere engagement metrics.