October 2024 Summaries

Sublime's Attack Spotlight series highlights real-world email threats, focusing on extortion attempts that exploit social engineering tactics to bypass language model-based phishing detectors. A notable attack involved novel text injection techniques, where attackers used command injections like "IGNORE EVERYTHING ELSE" to manipulate large language models (LLMs) by inducing them to ignore malicious content and focus on innocuous details, aiming to classify the email as legitimate. This method demonstrates the attackers' sophisticated understanding of LLMs' instruction-following tendencies, similar to other prompt injection tactics aimed at compromising security systems. Sublime detected this attack using a combination of signals, including extortion language, cryptocurrency references, and Cyrillic characters, and prevented it through a defense-in-depth approach using a Natural Language Understanding (NLU) model based on BERT, which is not susceptible to instruction manipulation.

Generative AI (GenAI) is increasingly used in email phishing attacks, enhancing their sophistication and accessibility for attackers who can create highly personalized phishing emails with minimal investment. This trend is evident in Business Email Compromise (BEC) and vendor impersonation campaigns, where AI-generated emails are detected through pattern recognition, sender verification, and GenAI detectors. However, identifying GenAI-written emails is not always effective due to attackers' ability to evade detection with simple modifications. Instead, a focus on indicators, contextual clues, and Tactics, Techniques, and Procedures (TTPs) is prioritized. Sublime employs a defense-in-depth strategy using Message Query Language (MQL) and natural language understanding (NLU) to detect phishing attempts by analyzing intent, tone, and context. The company has fine-tuned BERT, a context-aware language model, to enhance its NLU capabilities, allowing it to detect sophisticated, polymorphic email attacks by understanding language nuances. By generating synthetic email samples using GenAI, Sublime can build a diverse dataset to test and improve its detection models, making them more robust against evolving threats. DistilBERT, a variant of BERT, has demonstrated high accuracy and speed in detecting phishing attempts, outperforming other classifiers, making it suitable for large-scale email protection. This approach turns GenAI from a threat into a security asset, helping to detect and mitigate phishing attacks effectively.