When an AI agent should refuse to answer

The process of refusal in AI agents is crucial and challenging to engineer, as it is essential for preventing the dissemination of incorrect or harmful information. Four main scenarios necessitate refusal: when a query is out of scope, when there are permission issues, when the agent lacks confidence in its response, and when the request poses safety risks. Achieving effective refusal involves complex engineering to build proxy signals for confidence, such as grounding overlap and confidence calibration, and requires a user interface that provides context for refusals to maintain user trust. Effective refusal not only prevents misinformation but also enhances user trust by clearly acknowledging the user's intent, explaining the reason for refusal, suggesting alternative actions, and logging the interaction for future analysis. Ultimately, refusal is treated as a feature that strengthens the reliability of AI agents, emphasizing that an agent's trustworthiness is built on its ability to appropriately refuse to answer when necessary.