July 2025 Summaries

Promptfoo, a company focused on AI security, has successfully raised $18.4 million in a Series A funding round led by Insight Partners, with notable participation from Andreessen Horowitz. Founded in 2024, Promptfoo addresses the growing security challenges associated with deploying generative AI applications, offering solutions that cater to over 125,000 developers and more than 30 Fortune 500 companies. The company tackles issues like prompt injection, data leakage, and insecure tool use that arise from complex architectures such as agents, RAGs, and MCPs. With the addition of Ganesh Bell from Insight Partners to its board, Promptfoo aims to further its mission by enhancing support for agentic workflows, adopting MCP as a standard for tool calls, and integrating audio features for securing spoken applications. The company envisions a future where adversarial testing becomes as integral and routine as unit testing in traditional software development, positioning itself as a critical player in the AI security landscape. Promptfoo's development roadmap focuses on building robust AI security infrastructure akin to CI pipelines in DevOps, supporting its long-term vision of comprehensive and automatic security evaluations.

Black Hat and DEF CON USA 2025 are upcoming events where attendees can engage with AI security innovations and meet experts from various fields. The AI Summit, held at the Mandalay Bay Convention Center in Las Vegas on August 5, is a key event for discussing AI advancements, particularly focusing on how Fortune 500 companies manage LLM vulnerabilities. The Black Hat Business Hall will feature a talk by Promptfoo's CEO on August 6, showcasing how enterprises utilize Promptfoo for secure LLM deployment. Visitors can also book demos to see Promptfoo's capabilities or attend Arsenal Labs for practical demonstrations of security threats like prompt injection and data exfiltration. DEF CON, running from August 8-10, offers further engagement opportunities, including a community party at the Millennium FANDOM BAR and career discussions for roles like AI red teamer. Both events provide a platform for networking, learning, and exploring career opportunities in AI security.

The analysis of Grok 4, an AI developed by Elon Musk's xAI, reveals a complex political bias landscape that challenges initial expectations of right-wing propaganda. Contrary to media reports, Grok 4, along with other leading AI models like GPT-4.1, Gemini 2.5 Pro, and Claude Opus 4, leans left of center, with Grok 4 displaying a notably bimodal distribution that swings between extreme left and right positions, making it the most contrarian among its peers. The dataset used for this analysis, featuring 2,500 political questions, showed that Grok is unexpectedly critical of Musk's companies, suggesting a possible overcorrection for perceived bias. All tested models demonstrated a general left-leaning tendency on economic issues, reflecting the training data's influence rather than a deliberate ideological drive. The study highlights the inherent challenges of achieving true political neutrality in AI models and underscores the importance of transparency regarding AI biases.

AI red teaming focuses on simulating real-world attacks to identify vulnerabilities in artificial intelligence systems, diverging from traditional red teaming by emphasizing non-deterministic outcomes and the testing of AI models for issues such as toxicity, hallucinations, and data leaks. As AI systems become more integrated, the scope of AI red teaming includes testing not only the AI models themselves but also the broader system dynamics and interactions with plugins and agents, necessitating a cross-functional approach involving security engineers, ML specialists, and product teams. The evolution of red teaming practices can be structured in stages, from no testing to comprehensive AI assurance, with tools like Promptfoo enabling testing integration into development pipelines, thus fostering a culture of collaborative and continuous security assessment. Regulatory frameworks across regions, such as the EU AI Act and China's AI Measures, emphasize the need for robust AI security practices, making AI red teaming a critical component for compliance and risk management. Building a red teaming culture involves fostering cross-functional ownership, transparency, diverse perspectives, and incentives, ultimately leading to a trustworthy and resilient AI system that aligns with industry standards and earns user trust.

System cards are detailed documents accompanying the release of large language models (LLMs) that provide system-level information about the model's deployment, distinct from model cards, which focus solely on model details. Originating from Meta researchers and popularized by OpenAI in March 2023 with its GPT-4 release, these cards include comprehensive data on training, security policies, limitations, risk assessments, and model behaviors, serving as a critical resource for those concerned about application security and responsible AI practices. Existing examples from OpenAI and Anthropic highlight both the potential risks and the mitigations in place, such as red-teaming exercises and evaluations of cybersecurity and biological threats, while underscoring the importance of accountability and evolving standards as technology advances. The cards are seen as invaluable for educating users and informing deployment practices, suggesting that as system cards evolve, their format should maintain essential content while adapting to emerging technological concepts to enhance their utility and interest.

In the past year, breaches involving large language models (LLMs) have surged by 180 percent, prompting the need for a concise guide to the top security risks and mitigation strategies. The Open Worldwide Application Security Project (OWASP) has identified the top ten vulnerabilities for LLMs, including prompt injection, sensitive information disclosure, supply chain vulnerabilities, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. These issues can lead to unauthorized data access, misinformation dissemination, and resource overuse, among other risks. Mitigation actions include restricting model behavior, enforcing privilege control, validating inputs and outputs, limiting permissions, and conducting adversarial testing. The importance of educating users and developers, along with implementing content security policies and resource constraints, is emphasized to manage these vulnerabilities effectively. The comprehensive OWASP Top 10 for LLMs PDF provides further details and examples, serving as a valuable resource for navigating the complex security landscape of LLM applications.

The Model Context Protocol (MCP) adoption is rapidly increasing, with NPM installations reaching 4.7 million in July 2025, leading to the introduction of the Promptfoo MCP Proxy to manage security risks for enterprises using MCP servers. While MCP servers, which act as wrappers around existing APIs, are not inherently insecure, their use can create vulnerabilities, as seen in scenarios where malicious MCP servers exfiltrate sensitive company data. The security challenges arise from combining multiple tools in operational loops, which can result in unintentional data leaks. To mitigate these risks, developers are advised to use a single tool per loop, thoroughly review MCP servers, and ensure no conflicting actions occur during critical operations. The Promptfoo MCP Proxy addresses these challenges by offering enterprise-grade security controls, including whitelisting MCP servers, providing visibility and control over server combinations, and ensuring data protection, enabling organizations to safely leverage MCP servers for enhancing LLM capabilities without compromising security.

Promptfoo has achieved SOC 2 Type II compliance and ISO 27001 certification, underscoring its dedication to high standards in information security, privacy, and risk management. The company specializes in securing generative AI applications by identifying and addressing vulnerabilities through adversarial emulation and red teaming. These certifications demonstrate Promptfoo's robust internal controls and systematic approach to data protection, meeting the security and compliance requirements of modern enterprises. The company emphasizes that security is foundational, not just a checklist, and is committed to making large language models (LLMs) safer and more reliable for enterprises. Promptfoo invites community engagement through job applications, GitHub contributions, and participation in its Discord community, while expressing gratitude to its customers, community, and team for their support in reaching this milestone.

As organizations prioritize the security of machine learning models, two tools, Promptfoo's ModelAudit and Protect AI's ModelScan, have emerged to assist with identifying vulnerabilities. A comparison of these tools was conducted using 11 test files with known security issues, revealing significant differences in format support and detection capabilities. ModelAudit analyzed all 11 files, identifying 16 security issues, while ModelScan focused on pickle-based formats, detecting only 3 issues across 6 files. ModelAudit demonstrated broader format coverage, including configuration files and ONNX models, and provided more detailed vulnerability detection and risk scoring. In contrast, ModelScan excelled in pickle security scanning with a lightweight implementation suitable for environments using pickle-based models. Both tools offer command-line interfaces and JSON outputs for automation, but ModelAudit also includes a UI and additional security insights. Organizations are encouraged to evaluate their specific needs and possibly use both tools complementarily to address diverse ML framework requirements and security considerations.

As AI technology evolves, the importance of system prompt hardening grows to ensure the security and reliability of AI models against various attacks such as prompt injection, cache exploitation, and instruction overriding. Prompt hardening involves techniques like instruction shielding, syntax reinforcement, and layered prompting to maintain the integrity and intended behavior of system prompts, thus preventing unauthorized access and manipulation. For instance, instruction shielding prevents new instructions from overriding original prompts, while syntax reinforcement and layered prompting add complexity and structure to resist attacks. Evaluating the strength of these defenses can be accomplished through tools like Promptfoo, which enables testing and refining system prompts to withstand adversarial attempts. Ultimately, while external processing tools are crucial, robust system prompt hardening significantly contributes to secure AI interactions and reduces the burden on these tools.