July 2022 Summaries

Neeva, a private and ad-free search engine founded by former Google employees, emphasizes the importance of search quality measurement over user engagement metrics like clickthrough rate, which can inadvertently lead to irrelevant, clickbait content. Instead, Neeva employs human raters to evaluate the relevance of search results, particularly for technical programming queries, by comparing its performance to Google's. This evaluation involves a personalized approach, using a programming-specific rating team and tailored query sets from raters' personal browsing histories. Neeva's focus on rich user interfaces, such as providing direct code snippets in search results, often distinguishes it from Google, although Google's superior handling of long-tail queries is noted. The insights gained from these evaluations contribute to improving search algorithms, enabling offline experimentation, and forming high-quality datasets for machine learning training, ultimately aiming to optimize search engines for user satisfaction and relevance rather than mere engagement metrics.

Hugging Face's BLOOM, a multilingual large language model with 176 billion parameters, underwent an evaluation to assess its real-world application performance, revealing several challenges and limitations. Despite BLOOM being trained openly with contributions from over 1,000 researchers across 70 countries, its performance in human evaluations on tasks such as categorizing toxic speech, creative writing, question-answering, and marketing copywriting did not consistently match up to the expectations set by traditional academic benchmarks. The study highlighted the shortcomings of existing benchmarks, which often fail to capture the nuanced, creative, and practical abilities of language models, especially in areas like humor and serendipity. BLOOM's results in human evaluations indicated that while it showed potential in programming tasks, it struggled with consistency and accuracy in other areas, suggesting that the model's development could benefit from improved benchmarks and evaluation criteria that consider real-world applicability. The authors propose open-sourcing a dataset designed for human evaluation and establishing guidelines to better assess language model outputs, encouraging a collaborative approach to refining these models and their assessment methods.

Google's "GoEmotions" dataset, intended to classify 58,000 Reddit comments into 27 emotions, has been criticized for being significantly mislabeled, with 30% of the data reportedly erroneous. This mislabeling is attributed to issues such as the lack of contextual metadata for comments and the use of data labelers who may not be familiar with US-centric English idioms, culture, or sarcasm. The article highlights specific mislabeling examples, such as interpreting slang or sarcastic comments as negative emotions, which undermines the dataset's reliability for training machine learning models. The critique emphasizes the importance of high-quality data and suggests that Google treated data labeling as an afterthought, failing to consider the complexity and context required for accurate labeling. The article argues for a more sophisticated approach to data labeling, advocating for the involvement of culturally and contextually aware labelers and robust infrastructure to ensure the production of high-quality datasets, which are crucial for developing effective AI models.