June 2025 Summaries

As enterprises increasingly deploy AI applications at scale, red teaming has become crucial for identifying vulnerabilities, with Promptfoo and Microsoft's PyRIT emerging as prominent open-source tools in this space. Promptfoo is a developer-friendly toolkit that integrates directly into CI/CD pipelines, offering automated, context-aware security testing and actionable reports, making it suitable for continuous security checks and compliance reporting. It provides built-in tests for vulnerabilities like prompt injections and unauthorized tool usage, with features like visual dashboards and OWASP mapping. In contrast, PyRIT is a flexible Python framework designed for security researchers, allowing for the creation of custom red teaming scenarios with extensive scripting capabilities, making it ideal for those who prefer programmatic control and deep dives into AI-vs-AI attack orchestration. While both tools address core LLM security risks, Promptfoo emphasizes automation and ease of integration, whereas PyRIT offers a customizable and extensible approach, catering to different team needs and expertise levels.

As organizations deploy Large Language Models (LLMs) in production, identifying security vulnerabilities becomes crucial, leading to the development of two prominent open-source tools: Promptfoo and Garak. These tools offer distinct approaches to LLM red teaming, with Promptfoo focusing on dynamic, application-specific attack generation and Garak providing a curated library of known vulnerabilities. Promptfoo, used by companies like Shopify and Microsoft, excels in generating intelligent, context-aware attacks tailored to specific applications, particularly in complex Retrieval-Augmented Generation (RAG) and agent systems, and integrates seamlessly with CI/CD workflows. It offers an enterprise edition with features like on-premise deployment and team collaboration. Garak, developed with NVIDIA's support, focuses on validating against known vulnerabilities through a library of research-backed prompts and integrates with NeMo Guardrails. It is well-suited for security researchers looking to audit LLM endpoints for documented weaknesses. Both tools are actively maintained and cater to different aspects of LLM security, allowing teams to choose based on their specific needs for vulnerability discovery and compliance mapping.

Google's Gemini 2.5 Pro represents a notable advancement in multimodal AI, boasting capabilities such as extended context processing, multimodal vulnerabilities, enhanced reasoning, and function calling. These features present unique security challenges, prompting the use of tools like Promptfoo for systematic vulnerability testing through adversarial red teaming. The guide outlines procedures for setting up a testing environment, generating test cases, executing evaluations, and analyzing vulnerability reports, emphasizing the importance of thorough security testing before deployment. Security considerations include potential attack surfaces in extended token contexts, multimodal processing vulnerabilities, denial-of-service risks in enhanced thinking modes, and the real-world implications of function calls. The guide also covers benchmarking against other models and suggests custom test cases for Gemini-specific features. It provides insights into ensuring secure function-calling capabilities and compliance with security frameworks such as OWASP, NIST, and the EU AI Act. Users are encouraged to integrate regular testing and monitoring into their CI/CD pipelines to track and improve security over time.

Promptfoo has developed an advanced red teaming tool designed to bolster the security of Large Language Model (LLM) systems by simulating sophisticated attacks. The evolution of red teaming has progressed from simple static prompts to dynamically generated attacks that consider the context of the target application. Promptfoo's system enhances traditional penetration testing techniques by incorporating deep reconnaissance, strategic planning, adaptive attack execution, and persistent memory, allowing for a comprehensive understanding of the target system and enabling iterative exploitation strategies. The tool prioritizes attack vectors based on business impact and system vulnerabilities, ensuring that attacks are aligned with critical business objectives. By bridging the gap between LLM vulnerability research and practical application, Promptfoo aims to set a new standard for enterprise-grade LLM security, offering a system that continuously learns and adapts through a process of real-time monitoring and replanning.

Promptfoo, an open-source toolkit for systematic and measurable prompt engineering, has surpassed 100,000 users, reflecting its widespread adoption by developers, enterprises, and AI enthusiasts. Founded by Ian Webster and Michael D'Angelo, Promptfoo addresses the need for reliable AI application development through tools that evaluate model prompts and identify vulnerabilities via AI red teaming. The framework has expanded to support AI agents and multi-agent systems, offering specialized tools to test complex interactions to ensure safety and consistency. Industry recognition has been marked by integrations with platforms like OpenAI, AWS, and IBM, which feature Promptfoo in educational materials. Trusted by individual developers and Fortune 500 companies alike, the platform allows users to test prompts across various LLM providers, detect vulnerabilities, benchmark models, and integrate security testing into CI/CD pipelines. As Promptfoo continues to grow, it aims to democratize AI security and enhance its capabilities to meet the evolving needs of its community.

OpenAI's GPT-4.1 and GPT-4.5, with enhanced coding and instruction-following capabilities, pose unique security challenges that are explored through adversarial red teaming using Promptfoo. The guide emphasizes the importance of systematically testing these models to identify vulnerabilities, particularly due to their advanced long-context processing and literal interpretation abilities that could be exploited for malicious purposes. It outlines the setup process for creating a red teaming project, configuring the environment, and running evaluations to generate and execute test cases that probe for weaknesses. The guide also discusses comparing model variants, such as GPT-4.1 and GPT-4o, by using specific configurations to assess their security postures. Techniques for customizing test cases and ensuring compliance with security frameworks like OWASP and NIST are provided, alongside recommendations for regular testing, developing custom plugins, and integrating these evaluations into CI/CD pipelines to enhance security measures over time.