July 2024 Summaries

Promptfoo, a company specializing in AI security, has raised a $5 million seed round led by Andreessen Horowitz to enhance the security of AI applications for developers. The company addresses the growing security challenges and legal uncertainties faced by companies using large language models (LLMs) by focusing on fortifying the application layer where these models interact with the real world. Promptfoo offers the first pentesting product specifically designed for AI applications, simulating real-world threats to identify vulnerabilities such as context poisoning and tool misuse. Originating from the experiences of its founders in scaling AI and defending applications against threats, Promptfoo aims to empower developers by providing open-source tools that facilitate transparency and accountability in AI security. With over 25,000 software engineers from companies like Shopify and Amazon using their tools, Promptfoo is committed to establishing an open-source standard for AI security and continues to receive support from prominent industry figures and the open-source community.

The text discusses the automation of discovering jailbreaks in image models like OpenAI's Dall-E, enabling the generation of violent and disturbing images despite built-in safety measures. Using a process adapted from TAP, an Attacker-Judge reasoning loop modifies prompts to bypass the system's filters. The post provides examples of such jailbreaks across categories like violence, crime, harm, abuse, terrorism, massacres, accidents, and disasters, illustrating the potential for creating graphic content. It outlines a method using the promptfoo CLI tool to replicate these jailbreaks, which involves initializing a project, setting an OpenAI API key, and running evaluations to view jailbreaks through a web interface. The text notes the current method is simplified for speed and cost efficiency, with improvements anticipated in future OpenAI models to better prevent such jailbreaks.