December 2024 Summaries

The 2025 OWASP Top 10 list for Large Language Models (LLMs) has introduced "Unbounded Consumption" as a critical risk, replacing the previous focus on Model Denial of Service (DoS). This shift acknowledges the broader range of threats LLMs face, such as excessive and uncontrolled inferences that can lead to service degradation, financial losses, and even model theft. Unlike traditional DoS attacks that target network bandwidth, unbounded consumption exploits how AI models process requests, overwhelming systems with crafted prompts that drain computational resources. To mitigate these risks, organizations are encouraged to implement strategies such as rate limiting, input validation, resource management, and real-time monitoring. Tools and technologies like adaptive rate limiting, anomaly detection, and scalable infrastructure are essential for defending against these attacks. Regular audits, red-teaming exercises, and adherence to OWASP's guidelines further bolster security against evolving threats. The discussion highlights the importance of layered defenses to maintain service availability and protect against sophisticated attacks on LLMs.

Ensuring the safe handling of harmful content by large language models (LLMs) is crucial for production deployments, and the guide outlines the use of Promptfoo, an open-source tool, to conduct red team evaluations using the BeaverTails dataset. This dataset, designed by PKU-Alignment, provides test prompts across 14 harm categories such as discrimination, violence, and misinformation, to evaluate how well models manage harmful content. The evaluation process involves creating configuration files and running tests on models using Promptfoo, which can be configured for multiple LLM providers like OpenAI and Anthropic. The guide emphasizes the importance of testing models within the context of their application-specific configurations to identify where additional safety measures may be needed. It also highlights the need for regular testing, comparison across different models, and the combination of automated testing with human review to maintain and improve safety layers in AI systems.

Promptfoo is a tool used to run standardized cybersecurity evaluations against AI models, such as those from OpenAI, Ollama, and HuggingFace, to assess their vulnerability to prompt injection attacks. It allows testing not just on base models but also on applications that wrap these models, recognizing that behaviors can vary depending on implementation. Utilizing Meta's CyberSecEval benchmark, Promptfoo evaluates models across various languages and prompt injection techniques, providing reports on their ability to withstand such attacks. The process involves sending crafted prompts to models and using a judge LLM to assess whether the injection was successful. Even state-of-the-art models show significant vulnerability rates, highlighting the importance of regular testing and comparison across different models to ensure security. Promptfoo supports various providers and offers advanced configuration options, allowing users to customize tests or target specific applications. Regular evaluation, combined with human oversight and adherence to security best practices, is essential for maintaining secure AI systems.

The EU AI Act, effective from February 2, 2025, represents the first comprehensive legal framework aimed at regulating artificial intelligence systems, particularly those deemed high-risk, across sectors such as healthcare, education, and law enforcement. This regulation applies to all entities developing, using, importing, or distributing AI within the EU, with severe penalties for non-compliance, akin to GDPR. Article 5 of the Act delineates banned AI behaviors, including manipulation, exploitation based on personal traits, emotion inference in sensitive settings, and the use of biometric data for profiling. The text highlights potential misuse scenarios, especially with large language models (LLMs), and emphasizes the importance of implementing controls to prevent harmful AI behaviors. Promptfoo is introduced as a solution to test AI systems against prohibited behaviors by employing harmful generation plugins and custom policies to anticipate and counteract regulatory bypass attempts. Even though full compliance for high-risk systems is not required until August 2026, Promptfoo offers tools to ensure AI systems meet the EU AI Act’s cybersecurity, accuracy, and robustness standards.