The AI safety illusion: why current safety datasets fool us on model safety

AI models are often evaluated for safety using benchmarks that test their ability to refuse harmful requests, but recent research highlights significant shortcomings in these assessments. The study examines the quality of two widely used safety datasets, AdvBench and HarmBench, revealing that they do not accurately reflect real-world adversarial behavior due to their reliance on "triggering cues"—overtly negative or sensitive expressions designed to activate safety mechanisms. This reliance leads to inflated safety evaluations, as models appear safe when they are merely responding to these cues rather than resisting genuine malicious intent. The research introduces the concept of "intent laundering," a technique that removes triggering cues while preserving malicious intent, demonstrating that models considered safe often fail when these cues are absent. This exposes a gap between current safety evaluations and real-world threats, suggesting that AI safety research must develop better benchmarks and alignment techniques to more accurately model harmful behavior and improve the robustness of AI models against realistic misuse scenarios.