November 2024 Summaries

This guide outlines the process of using Promptfoo for adversarial testing, or red teaming, to evaluate the safety and security of models hosted on Ollama, with Llama 3.2 3B as an example. It provides step-by-step instructions for setting up the testing environment, configuring the Ollama model as the target, and specifying the purpose and plugins for vulnerability testing. The guide emphasizes the importance of defining high-quality purposes to generate effective adversarial tests, and it includes plugins to test for issues such as harmful content, PII leakage, false information, and model impersonation. Strategies like jailbreak and composite jailbreak are used to deliver adversarial inputs, and the testing process involves generating and running test cases, grading responses, and analyzing vulnerabilities through a detailed report. The guide also highlights common vulnerabilities in Llama models, such as prompt injection and harmful content generation, and suggests mitigation strategies like adding safety constraints, input validation, and output filtering.

The guide provides a comprehensive walkthrough for using Promptfoo to conduct adversarial testing, or "red teaming," on HuggingFace models to identify vulnerabilities. It details the setup process, including the installation of Node.js, acquisition of a HuggingFace API token, and initialization of a project using Promptfoo. The guide explains how to configure the HuggingFace provider and red teaming parameters in a promptfooconfig.yaml file, focusing on testing the Mistral 7B model for text generation with specific configurations like temperature and token generation limits. Key components include defining the number of tests, the purpose of the model, plugins for vulnerability types, and strategies for adversarial input. The process involves generating test cases, running them against the model, and analyzing results through reports that categorize vulnerabilities, assess their severity, and suggest mitigations. The guide emphasizes the importance of re-evaluating the model after implementing changes to ensure vulnerabilities are addressed effectively.

Promptfoo has introduced a new strategy, GOAT, designed to jailbreak multi-turn conversations in AI models, inspired by Meta's research on agentic red teaming systems. Unlike traditional single-turn attacks, GOAT uses a multi-turn approach where an attacker language model (LLM) engages in ongoing dialogue with a target model, utilizing a structured three-step process: observation, thought, and strategy. This iterative process allows the attacker LLM to dynamically adapt its techniques, simulating human-like adversarial interactions to uncover vulnerabilities in AI models over extended conversations. The GOAT strategy leverages a customizable toolbox of red teaming techniques, such as priming responses, hypotheticals, and persona modifications, to effectively bypass safety mechanisms and expose weaknesses that static methods may miss. By simulating real adversarial behavior and adapting strategies throughout the interaction, GOAT provides a more effective way to test the resilience of LLMs, particularly in conversational AI applications like chatbots and agentic systems.

Data poisoning is a significant security threat targeting AI systems, particularly those using Retrieval-Augmented Generation (RAG), by corrupting the external knowledge base these systems rely on for accurate information. This attack allows malicious actors to inject harmful content into databases, leading AI systems to generate incorrect or harmful outputs, which can have severe consequences in sectors like healthcare, finance, and security. Just a few strategically crafted documents can manipulate AI responses with high success, exploiting the AI's reliance on external context. The attacks can take various forms, such as instruction injection, context poisoning, and retrieval manipulation, often bypassing traditional security measures due to their sophisticated nature. The proliferation of RAG architecture introduces new vulnerabilities, including permission bypass, authentication gaps, and regulatory risks, as AI systems can inadvertently expose sensitive data. Real-world examples demonstrate the efficacy of these attacks, such as the Microsoft 365 Copilot exploit and ChatGPT memory poisoning, which show how attackers can gain unauthorized access and persistently exfiltrate data. Mitigation strategies include deterministic access control, input filtering, embedding analysis, and response filtering to detect and block malicious content, ensuring AI systems remain secure and reliable.