We're Afraid Language Models Aren't Modeling Ambiguity - Summary

The paper explores the critical role of managing ambiguity in natural language understanding and assesses the capability of language models to identify and differentiate potential meanings. It introduces AMBIENT, a linguist-annotated benchmark consisting of 1,645 examples showcasing various ambiguities, and uses it to create tests for evaluating pretrained language models, revealing that recognizing ambiguity is still a significant challenge even for advanced models like GPT-4. Furthermore, the study includes a case study demonstrating how a multilabel natural language inference (NLI) model can effectively detect misleading political claims in real-world scenarios, underscoring the importance of handling ambiguity for the success of language models.