November 2025 Summaries
138 posts from Lakera
Filter
Month:
Year:
Post Summaries
Back to Blog
David Haber's session at the 2024 RSA Conference highlights the transformative impact of AI on cybersecurity, emphasizing the unique challenges posed by AI systems that traditional measures fail to address. He introduces Gandalf, a viral prompt injection game developed by Lakera, which simulates real-world attacks on AI to expose vulnerabilities and develop new security strategies. As players engage with the game, they encounter increasingly complex scenarios designed to extract sensitive information, thereby gaining insights into AI vulnerabilities. Major corporations, hacker communities, and universities utilize Gandalf to better understand and safeguard AI systems. The session underscores the importance of gamifying AI red teaming as a method for crafting effective security measures tailored to the evolving nature of AI technologies.
Nov 28, 2025
441 words in the original blog post.
At the OWASP Global AppSec DC 2025, AI security emerged as a central focus, with discussions and presentations highlighting the challenges and opportunities it presents to application security. The event, marked by a vibrant and futuristic atmosphere, emphasized the need for securing AI systems already in place, particularly addressing agentic behavior, which introduces new failure modes and security dynamics. Daniel Miessler's keynote on Claude Code underscored AI's potential to enhance productivity while also posing security risks, a theme that resonated throughout the conference sessions and informal conversations. The Breaker Track became a hub for candid technical exchanges, with participants sharing real-world incidents and the effectiveness of offensive AI work. The event underscored the urgency of integrating AI security into existing frameworks, with the community keenly aware of the evolving threat landscape and eager to find solutions.
Nov 28, 2025
1,502 words in the original blog post.
The new Model Context Protocol (MCP) specification, set to be released on November 25, 2025, represents a significant evolution in the development and security of agentic AI systems. This update addresses the growing complexity and risks associated with AI agents by introducing identity verification for MCP servers, mandatory authentication protocols, and support for asynchronous tasks, which transform MCP into a more sophisticated workflow engine. By integrating HTTP transports and streaming, the new spec enhances observability, though it also complicates correlation efforts. Additionally, the introduction of a registry system for MCP servers poses new supply-chain risks, requiring developers and security teams to monitor server identities, validate authorization flows, and track long-running tasks more diligently. As MCP shifts from a simple integration tool to critical infrastructure, it necessitates a strategic evolution in security practices to ensure agents remain secure and aligned with organizational goals.
Nov 28, 2025
1,882 words in the original blog post.
Indirect Prompt Injection (IPI) is a significant vulnerability in modern AI systems, where attackers embed hidden instructions in data sources that AI models consume, such as webpages, PDFs, and emails, rather than directly interacting with the models through visible prompts. IPI exploits the AI's tendency to treat all ingested text as meaningful, which can lead to unintended actions, data leaks, or system compromises, especially when models are integrated with agentic capabilities that allow them to browse, retrieve, write, or execute tasks. The challenge in mitigating IPI lies in the inherent architectural design of AI systems, which blend trusted and untrusted inputs into a single context stream, making it difficult for models to distinguish between legitimate instructions and malicious ones. Effective mitigation requires a systems-level approach, including implementing trust boundaries, context isolation, output verification, and strict validation of tool interactions. By treating all external data as untrusted and reinforcing these defenses, organizations can better protect against the evolving threat posed by IPI, which continues to escalate as AI systems become more autonomous and integrated with external content.
Nov 28, 2025
4,189 words in the original blog post.
Recent advancements in computer vision have sparked interest in their application to medical imaging, offering significant potential to enhance patient care by expediting processes like disease screening and supporting complex diagnostic tasks. However, transitioning from prototype to production in medical imaging systems poses substantial challenges, particularly concerning model robustness in variable practical conditions. Despite strong performance metrics, models often exhibit vulnerabilities when faced with real-world factors such as patient movement and diverse equipment setups, which necessitate comprehensive machine learning testing and robustness analysis. A case study by the Lakera team highlights these issues, revealing that even state-of-the-art models experience critical failures under certain conditions, underscoring the need for thorough testing and improvement before deployment in clinical settings.
Nov 16, 2025
1,376 words in the original blog post.
Lakera, a company specializing in AI security, has secured $20 million in funding to enhance the safety of business generative AI applications by preventing them from malfunctioning and exposing sensitive data. The team has played a significant role in advancing Dropbox's journey in generative AI and offers expert-recommended security policies that users can apply swiftly to protect their AI applications. The company provides resources and regular updates on its products and developments in the AI and large language model (LLM) sectors, inviting users to subscribe for the latest information.
Nov 16, 2025
236 words in the original blog post.
By 2025, research on large language model (LLM) hallucinations has evolved to view them as a systemic incentive problem, with models being trained to prioritize confident guessing over calibrated uncertainty due to existing training objectives and evaluation metrics. This reframing has led to new mitigation strategies, including calibration-aware rewards, targeted fine-tuning, retrieval-augmented generation with span-level verification, and internal detection mechanisms. Despite these advances, hallucinations persist, particularly in low-resource languages and multimodal tasks, underscoring the importance of designing systems that manage uncertainty transparently and predictably rather than aiming for an impossible eradication of errors. The shift in focus from eliminating hallucinations to managing uncertainty reflects a broader understanding that while perfect reliability may be unattainable, enhancing transparency and predictability in model outputs is crucial for maintaining trust in AI applications.
Nov 16, 2025
2,342 words in the original blog post.
Agent Breaker is a GenAI security platform designed to model real-world vulnerabilities in AI systems through a series of gamified challenges that double as research testbeds. Created by Lakera, it simulates various attack scenarios on mock agentic AI applications, such as data poisoning and prompt injections, to expose inherent weaknesses and test defenses. The platform includes ten levels of increasing difficulty, each reflecting realistic threat landscapes, and is scored using a comprehensive set of metrics that measure the success and impact of different attack vectors. It serves as both an educational tool for security engineers and a competitive arena for red teamers, offering a leaderboard system across various foundational models like GPT-4 and Claude 3.7. By providing a controlled environment for stress-testing AI applications, Agent Breaker aims to generate actionable data that informs better security measures and guardrails, while also fostering a global community of over 100,000 participants who contribute to the continuous improvement of GenAI security practices.
Nov 16, 2025
2,261 words in the original blog post.
The AI Risk Map is a detailed guide designed to help individuals and teams navigate the complex landscape of AI security, offering insights into frameworks, threats, and risks associated with AI technologies. It serves as a comprehensive resource for professionals, security enthusiasts, and those curious about AI, highlighting the vulnerabilities of large language models (LLMs) and providing practical security advice, such as data sanitization and PII detection. The guide introduces Lakera's tools like the Gandalf education game and the Lakera Guard security solution, which are designed to enhance understanding and counteract common AI threats. It maps various AI risk frameworks, including MITRE ATLAS, OWASP's Top 10 for LLMs, and the NIST AI RMF, elucidating their overlaps, differences, and coverage of the GenAI lifecycle stages. Additionally, it presents a curated list of the Top 25 AI Risks, linking them to specific lifecycle stages and frameworks to aid in prioritization and understanding. The guide is created for security engineers, product managers, compliance teams, and decision-makers, aiming to clarify the fragmented landscape of AI risk management and ensure focus on the most pertinent risks.
Nov 16, 2025
407 words in the original blog post.
Choosing the best machine learning models for computer vision involves assessing various factors beyond standard metrics like precision, recall, and RMSE. The complexity arises because these metrics can obscure crucial information about model performance in specific contexts, such as different customer sites or image variations. Developers should employ subgroup analysis to ensure models perform consistently across different conditions, and robustness testing to gauge how models handle input variations. It's also vital to consider model biases and fairness, especially when these could affect user experience or safety. Production constraints, like memory footprint and inference time, can influence the model choice as they may diverge from performance observed in controlled environments. The use of tools like MLTest can help manage these complexities by providing a comprehensive view of model performance, including robustness, biases, and operational metrics, thereby enabling better decision-making in model selection.
Nov 16, 2025
1,412 words in the original blog post.
Lakera has announced updates to their Lakera Guard content moderation tool, enhancing its ability to detect and prevent violent, dangerous, and illicit content to ensure AI applications remain secure and compliant. The update improves detection across categories like violence, self-harm, illicit activities, and discussions of firearms and dangerous weapons, all while maintaining performance efficiency with minimal latency. The customizable nature of the new detectors allows users to tailor the moderation to their specific needs, offering a robust safety net for both public-facing and enterprise AI platforms. These advancements are part of Lakera's broader efforts to enhance AI security, with the company also offering expert-recommended policies to further protect AI applications.
Nov 16, 2025
534 words in the original blog post.
A recent study highlights a zero-click remote code execution exploit that leverages agentic Integrated Development Environments (IDEs) such as Cursor, demonstrating how attackers can use common AI coding assistants to execute malicious instructions without user interaction. The attack exploits the Model Context Protocol (MCP) and integrations like Google Docs, allowing attackers to silently share documents with victims, which the AI assistant then processes and executes, leading to credential theft and persistent system access. This vulnerability arises not from a patchable bug but from the inherent functionality of agentic workflows and MCPs, turning them into potential entry points for large-scale organizational attacks. The research emphasizes the need for layered defenses, including robust guardrails, cautious allow lists, and hardened configurations, to prevent such exploitations and secure AI-driven environments.
Nov 16, 2025
2,431 words in the original blog post.
Lakera has launched an innovative AI Model Risk Index, which has been highlighted by Yahoo Finance for its potential to establish a new standard in evaluating the security of large language models (LLMs). The Index aims to address increasing concerns over AI model vulnerabilities by offering enterprises a practical and clear framework for risk assessment. The announcement also mentions Lakera's role in expediting Dropbox's journey with Generative AI (GenAI) by providing expert-recommended security policies that can be quickly implemented and adjusted as needed. Lakera encourages interested parties to engage with its offerings through guides and newsletters, keeping them informed about the latest developments in AI LLM technology.
Nov 16, 2025
278 words in the original blog post.
Lakera is actively involved in enhancing the security of generative AI applications by aligning with the OWASP Top 10 for LLMs (2025), a key framework for identifying and mitigating risks in AI systems. The 2025 edition highlights the importance of addressing vulnerabilities throughout the AI lifecycle, from training to deployment. Lakera contributes to this effort through its Lakera Red and Lakera Guard solutions; Lakera Red focuses on simulating real-world attacks to identify risks during development, while Lakera Guard provides real-time protection against threats at runtime. These tools help secure AI applications by detecting and mitigating prompt injections, data leaks, and other vulnerabilities, although some areas, such as supply chain risks, remain partially addressed. By operationalizing OWASP standards, Lakera ensures that generative AI applications are not only secure and compliant but also trustworthy, thereby supporting the industry's leading security practices.
Nov 16, 2025
2,050 words in the original blog post.
Fuzz testing is a technique traditionally used in software testing that provides invalid, unexpected, or random data to programs to identify potential software bugs and vulnerabilities. This method is now being adapted for machine learning (ML) systems to uncover robustness issues during development. Fuzz testing is particularly useful for computer vision applications due to their expansive input space and potential for subtle bugs. It helps identify problematic inputs by generating synthetic data that the system may not have encountered during training, thus improving system reliability. Techniques like DLFuzz and DeepHunter mutate input images to trigger failures by activating rarely used neurons or preserving image labels through transformations. Fuzz testing aids in stress testing ML systems, ensuring they perform well under expected conditions and fail gracefully when presented with challenging inputs. This approach can highlight areas where further data augmentation or collection is needed, and it is recommended as part of a comprehensive testing suite for ML components to enhance resilience against unexpected scenarios.
Nov 16, 2025
1,856 words in the original blog post.
Lakera has announced it raised $20 million in a Series A funding round led by Atomico, with participation from Citi Ventures, Dropbox, and existing investors, bringing its total funding to $30 million. This investment underscores the growing urgency to secure Generative AI (GenAI) applications against AI-specific risks such as prompt and data poisoning attacks, which can exploit models to take unintended actions. Lakera's approach uses proprietary AI to provide real-time security for GenAI applications, ensuring robust protection without sacrificing user interactivity. Their technology allows for centralized control through a single API call, enabling security teams to set application-specific policies without altering existing models. The company's unique position is bolstered by Gandalf, an AI educational game that serves as the world's largest AI red team, generating extensive threat data that informs Lakera's research and development. With plans to expand its presence in the US and Europe, Lakera aims to accelerate product development and market reach, reinforcing its mission to secure GenAI applications safely and effectively.
Nov 16, 2025
909 words in the original blog post.
Lakera has introduced a new feature in its Policy Control Center called custom regular expression (regex) detectors, enhancing AI security by allowing users to define specific words, text strings, rules, and patterns to meet unique security and content moderation needs. This feature is integrated into Lakera Guard, enabling the creation of customized content moderation and sensitive data screening detectors without code changes, and is applicable in various sectors like e-commerce, finance, healthcare, and international business to prevent data leakage and ensure compliance with regulations. The custom regex detectors work alongside native detectors for combined analysis, providing businesses with flexibility and control over their security measures, and are part of Lakera's ongoing commitment to improve security offerings and keep organizations ahead of evolving threats. Lakera plans to showcase these capabilities in a live session on October 15th, 2024, offering insights into customizing GenAI security controls and demonstrating how enterprises manage and scale security across diverse AI use cases.
Nov 16, 2025
789 words in the original blog post.
AI chatbots have become integral to modern business operations, providing cost-effective, efficient customer service and transforming areas like sales, marketing, and HR. Initially limited in capability, chatbots have evolved significantly with the advent of large language models, now offering human-like interactions and personalized customer experiences. Businesses such as Sephora, LambdaTest, and Bank of America have successfully utilized chatbots to enhance customer engagement and operational efficiency. Despite their benefits, chatbots carry risks such as security breaches, misinformation, and inherent biases, necessitating robust security measures. Future advancements in natural language processing and integration with technologies like voice assistants are expected to further refine chatbot capabilities, offering even more immersive and personalized customer interactions.
Nov 16, 2025
2,261 words in the original blog post.
Lakera, a leading AI security company, has announced its achievement of SOC 2 Type I compliance for its AI security software, Lakera Guard, demonstrating a commitment to maintaining high standards of data security and privacy. This compliance, validated by Prescient Assurance in accordance with the American Institute of Certified Public Accountants (AICPA) standards, affirms Lakera's dedication to secure development, personnel, and testing processes, as well as cloud security. Lakera utilizes AWS's infrastructure for optimal security and availability and continues to focus on advancing AI security by leveraging expert-recommended policies to protect customer data effectively. This achievement marks a significant milestone in Lakera's ongoing efforts to lead in AI security and ensure the protection of its clients' information.
Nov 16, 2025
643 words in the original blog post.
Lakera has announced a significant update to its Lakera Guard security platform by introducing the new Policy Control Center, which allows security teams to manage AI security policies organization-wide without needing to alter existing code. This update enhances real-time control and integration ease, offering tailored protection and adaptable defenses for AI applications. Lakera Guard addresses challenges posed by the integration of generative AI into organizational infrastructure, providing instant control over security measures, the ability to customize defenses, and a user-friendly interface for quick adjustments. Built to evolve with AI advancements, Lakera Guard ensures comprehensive protection while simplifying implementation, enabling organizations to focus on innovation with confidence in their AI security measures. A live session on October 15th, 2024, will showcase the Policy Control Center's capabilities, demonstrating how it allows enterprises to customize security controls across diverse AI applications.
Nov 16, 2025
937 words in the original blog post.
Lakera has co-published an article in a Nature journal focusing on testing methodologies for AI in medical imaging systems, emphasizing the importance of reliability and regulatory approval before such technologies are used in hospitals. The paper provides recommendations on collecting test datasets in pathology and medical imaging, addressing key questions about the number and type of images needed, managing low-prevalence subsets, detecting bias, and understanding regulatory requirements across countries. Lakera's MLTest product aims to integrate reliability into engineering processes for medical imaging companies, expediting regulatory approval and ensuring safe innovation in the medical domain. The company also highlights its contributions to Dropbox's GenAI journey and offers security solutions for GenAI applications.
Nov 16, 2025
543 words in the original blog post.
The rapid evolution of Large Language Models (LLMs) has significantly transformed the landscape of AI-driven systems, with numerous models emerging that offer various features and capabilities. This comprehensive guide highlights the leading LLMs such as GPT-4, Gemini, LLaMA, Claude, Aya, and BLOOM, each bringing unique advancements in understanding and generating human-like text across different contexts and languages. These models have been developed to cater to diverse applications, including content creation, customer service, coding assistance, and more, with an emphasis on safety, accessibility, and inclusivity. As the field continues to expand, choosing the right LLM involves considering not only performance but also the potential risks and security measures associated with each model. Open-source initiatives and collaborative platforms like Hugging Face play a crucial role in democratizing access to these advanced AI tools, fostering innovation and enhancing human-AI collaboration. The ongoing development of models like OpenAI's GPT-5 and Meta's LLaMA 3 indicates a future where AI systems become increasingly versatile, reliable, and aligned with human values, promising a more interconnected and intelligent world.
Nov 16, 2025
4,694 words in the original blog post.
Lakera has introduced Lakera Guard, a developer-first API designed to provide enterprise-grade security to applications utilizing Large Language Models (LLMs), addressing new security challenges such as prompt injection, sensitive information leakage, and hallucinations. This API acts as a protective shield, ensuring any communication in and out of an LLM passes through its security checks, akin to a network firewall, and can be integrated into existing application stacks with minimal effort. Lakera Guard delivers out-of-the-box protection and continuously evolves its security intelligence by drawing from public sources, community data, and the latest research, with its proprietary vulnerability database expanding rapidly. Currently in private beta with limited access, Lakera Guard allows developers to maintain control over application flow and enhances security by providing detailed JSON findings and enabling customizable security responses to identified threats, ensuring safe and secure LLM operations for organizations worldwide.
Nov 16, 2025
1,232 words in the original blog post.
Lakera has announced its integration with FiftyOne by Voxel51, enhancing the capabilities for evaluating and testing computer vision models. This partnership allows users to leverage FiftyOne's visualization tools to gain deeper insights into model predictions generated by Lakera's MLTest, facilitating easier model comparisons and selection. MLTest provides advanced functionalities that go beyond traditional metrics, including robustness and fairness assessments, and allows for detailed diff-like comparisons between models. The integration further enables users to conduct granular prediction-by-prediction analyses and deep failure analyses by clustering images based on model performance discrepancies. The setup process is user-friendly, with documentation and examples available to streamline onboarding.
Nov 16, 2025
652 words in the original blog post.
As AI systems become integral to modern infrastructure, traditional cybersecurity approaches are proving inadequate to address the unique challenges posed by AI, leading to a need for a new security paradigm. Unlike static systems protected by conventional cybersecurity, AI systems are dynamic, complex, and difficult to predict, resembling black boxes that even their creators struggle to fully understand. This evolving landscape requires innovative security solutions tailored to AI's unique characteristics, as attackers increasingly exploit these systems using creativity rather than advanced technical skills. The text underscores the danger of relying on legacy security tools wrapped in AI enhancements, which can create a false sense of security, and emphasizes the need for a foundational rethinking of AI security strategies. The future of cybersecurity hinges on developing new tools and methodologies designed specifically for AI, paralleling the significant shift seen during the transition to cloud-based security solutions.
Nov 16, 2025
1,076 words in the original blog post.
Lakera's AI security solutions have been highlighted in a recent Help Net Security feature, focusing on how organizations like The Motley Fool are safely scaling generative AI. The article emphasizes the importance of establishing robust security measures, such as usage mapping, automated testing, and continuous monitoring, when deploying large language models (LLMs) at scale. The Motley Fool utilizes Lakera's tool, Lakera Red, to stress-test LLMs for vulnerabilities, underscoring the necessity for strict security practices akin to those used for critical applications. This recognition underscores Lakera's pivotal role in aiding enterprises to build secure, resilient AI systems equipped to handle complex, real-world challenges.
Nov 16, 2025
356 words in the original blog post.
The text explores the concept of visual prompt injections within AI models, particularly focusing on GPT-4V's capabilities and vulnerabilities. Visual prompt injections involve embedding malicious instructions within images, which the model can interpret and act upon, often leading to unintended actions. Examples from Lakera's hackathon demonstrate how simple text prompts on images can manipulate the model's behavior, such as acting as invisibility cloaks or altering image descriptions. The text highlights the potential risks and challenges of integrating multimodal AI systems, emphasizing the need for robust security measures, as malicious actors could exploit these vulnerabilities. Lakera is actively developing solutions, including a visual prompt injection detector, to mitigate these threats, while promoting their Lakera Guard security tool to protect against such vulnerabilities in AI-powered applications.
Nov 16, 2025
1,470 words in the original blog post.
The blog post by Mateo Rojas-Carulla explores the intricacies of model selection and robustness testing in computer vision, focusing on how models with similar mean Average Precision (mAP) scores can behave differently in production. It highlights the importance of robustness analysis over aggregate test metrics, using Lakera's MLTest to identify vulnerabilities and differentiate between models that might seem identical based on mAP alone. The post emphasizes that augmentation strategies, while crucial for building robust models, must be carefully tested as they can sometimes degrade performance rather than improve it. By conducting a detailed robustness scoring, developers can ensure that models are better prepared for the challenges of real-world deployment, making MLTest an essential tool in the model development workflow on platforms like Roboflow.
Nov 16, 2025
1,906 words in the original blog post.
Lakera's new approach to GenAI security policy management focuses on providing pre-built, expert-recommended policies that allow security teams to quickly deploy protections for LLM-powered applications, such as those used by Dropbox, while maintaining flexibility for future customization. The platform offers one-click policies tailored to common deployment scenarios, such as public-facing applications or internal tools, with a global sensitivity setting to match specific risk thresholds. As security needs evolve, teams can progressively introduce more complex controls without starting from scratch, supporting a natural progression from basic to advanced security measures. This update aims to streamline the implementation of AI security, leveraging lessons learned from early adopters to offer clear, scalable policy defaults that align with a company's risk appetite and maturity level.
Nov 16, 2025
1,019 words in the original blog post.
The text discusses the transformative impact of the Internet of Agents (IoA), a network of AI-to-AI applications that is anticipated to redefine the cybersecurity landscape. As AI technologies like Large Language Models (LLMs) become more integrated into everyday life, they are expected to enhance human productivity and creativity by taking on more autonomous roles, such as negotiating contracts or writing software. However, this rapid adoption introduces significant cybersecurity challenges, as the interconnectedness of AI agents increases the risk of manipulation and exploitation. The text emphasizes the need for a new cybersecurity paradigm that incorporates containment, traceability, and accountability to manage the inherent risks of the IoA. This approach is necessary to ensure that the benefits of AI advancements can be realized without compromising security, highlighting the urgency for innovative security measures to be developed in parallel with AI technologies.
Nov 16, 2025
1,553 words in the original blog post.
Data exfiltration, a significant cybersecurity threat, involves the unauthorized transfer of data from computers or devices, often exacerbated by the rise of artificial intelligence (AI). AI presents a dual challenge in cybersecurity, as it can both prevent and facilitate sophisticated data theft methods. Understanding the difference between data leakage, breaches, and exfiltration is crucial, as each involves varying levels of data exposure and theft. Common methods of exfiltration include phishing, malware, insider threats, and AI-powered techniques such as deepfake and model manipulation. Detection and prevention require a combination of traditional methods like network monitoring and advanced AI-based solutions for behavior analysis and predictive security. High-profile cases, such as those involving Tesla and General Electric, highlight the real-world impact of data exfiltration, underscoring the need for comprehensive strategies that integrate AI technologies with conventional security measures to safeguard sensitive information.
Nov 16, 2025
5,643 words in the original blog post.
A vulnerability identified as CVE-2025-59944 was discovered by Lakera researcher Brett Gustafson in Cursor, an AI-driven IDE that allows developers to write code using natural language, highlighting the potential risks associated with agentic developer tools. The flaw involved a case-sensitivity bug that enabled attackers to bypass file protections, potentially allowing unauthorized modifications to configuration files and leading to remote code execution. Cursor's protection logic, which matched filenames case-sensitively, failed on case-insensitive filesystems like those on Windows and macOS, allowing crafted inputs to exploit this loophole. Although the issue was swiftly addressed in Cursor 1.7 by normalizing file paths and comparing them without case sensitivity, the incident underscores the evolving challenges in software security as AI tools gain more autonomy. This case illustrates the necessity for security measures to focus on understanding intent rather than just syntax, as automation expands the attack surface of development environments, necessitating a shift from static validation to behavioral and contextual defense.
Nov 16, 2025
1,231 words in the original blog post.
Testing machine learning (ML) systems is essential, particularly in mission-critical contexts, where reliability and compliance are crucial. While companies like Tesla have advanced testing frameworks, smaller teams can also adopt effective ML testing strategies. These include specifying an operational domain to establish system reliability, stress-testing to evaluate robustness using techniques like fuzz testing and metamorphic relations, and employing regression sets to ensure iterative performance improvements. By implementing these strategies, smaller teams can achieve high returns on investment in developing reliable ML systems. Lakera's MLTest offers a tool to identify performance vulnerabilities in computer vision systems, enhancing reliability for development teams.
Nov 16, 2025
715 words in the original blog post.
Lakera announced its participation in the AI House Davos event, held alongside the World Economic Forum 2024, where its CEO, David Haber, will lead a session titled "AI Safety Unplugged: Navigating the Risks Without the Hype." This interactive panel will feature notable figures like Yann LeCun from Meta and Max Tegmark from MIT, focusing on critical AI topics such as the advancements towards artificial general intelligence, the proliferation and alignment of intelligent agents, safety concerns surrounding generative AI, and AI governance best practices. The session aims to address pressing issues related to AI security and governance, with contributions from leaders in AI research, development, and policy-making. Lakera positions itself as a leading developer-first AI security platform, offering a comprehensive AI vulnerability database to support secure AI product development.
Nov 16, 2025
602 words in the original blog post.
Personally Identifiable Information (PII) is becoming increasingly difficult to protect in the era of Generative AI (GenAI), as traditional data protection methods prove insufficient. Unlike past approaches that focused on encrypting and securing stored data, GenAI systems can inadvertently generate, infer, or expose PII in their outputs, creating new vulnerabilities. This shift challenges legacy Data Loss Prevention (DLP) tools, which rely on pattern recognition and fail to detect nuanced, multilingual, or contextual disclosures. Lakera Guard offers a real-time solution by analyzing model interactions to identify both direct and indirect PII across languages, adapting to the complexities of natural language. The necessity for real-time, context-aware defenses is highlighted by incidents like the Samsung source code leak and findings that a significant portion of AI prompts contain sensitive information. As PII increasingly resides in language rather than fixed formats, security approaches must evolve to monitor and mitigate these new risks.
Nov 16, 2025
2,215 words in the original blog post.
The 2025 GenAI Security Readiness Report by Lakera highlights the rapid adoption of generative AI across enterprises, emphasizing a disconnect between the speed of implementation and the maturity of security measures. Despite nearly half of organizations using GenAI solutions, only 19% express high confidence in their security posture, with significant concerns about vulnerabilities like prompt injection attacks and data leakage persisting. The report identifies a skills gap, with 39% of respondents citing a shortage of skilled AI-security talent, compounded by the complexity of integrating GenAI systems. Privacy, once the top concern, has been surpassed by emerging threats such as adversarial misuse and offensive AI, necessitating a shift from compliance-based protection to more dynamic, runtime confrontations. Through insights from industry professionals, the report underscores that readiness for GenAI security is an ongoing process rather than a fixed destination.
Nov 16, 2025
750 words in the original blog post.
Lakera, a leading real-time Generative AI security company, has secured $20 million in a Series A funding round led by Atomico, with participation from Citi Ventures, Dropbox Ventures, and existing investors, boosting its total funding to $30 million. As enterprises increasingly adopt Generative AI applications, which are predicted to be used by 80% of enterprises by 2026, Lakera addresses the unique cybersecurity risks these technologies pose. Traditional cybersecurity measures are inadequate against threats such as prompt attacks and AI-targeted worms, prompting a demand for Lakera's innovative solutions that protect without compromising user experience. Their API, easily integrated with a single line of code, provides low-latency security for any GenAI model. Lakera's AI educational platform, Gandalf, which boasts over a million users, contributes to their robust threat database, allowing continuous updates to their security software. The company plans to use the new funding to enhance product development and expand its US presence, capitalizing on its success with clients like Dropbox and a top US bank. Sasha Vidiborskiy of Atomico will join Lakera's board, endorsing the company's thought leadership and expertise in AI cybersecurity.
Nov 16, 2025
1,273 words in the original blog post.
Lakera, a security platform, has announced the launch of its new hacking simulator game, Gandalf: Agent Breaker, which builds on its original game, Gandalf, by offering a more immersive experience that allows players to interact with ten GenAI applications to learn about agentic security. The game is now available globally, and it coincides with Lakera's efforts to enhance Dropbox's GenAI journey by providing expert-recommended security policies for GenAI applications. The company also offers resources such as guides and newsletters for those interested in staying updated on AI LLM advancements and Lakera's product developments.
Nov 16, 2025
281 words in the original blog post.
Machine learning models often rely on validation sets to choose the best-performing model, but this approach can be misleading, as validation accuracy does not necessarily predict real-world performance due to data distribution shifts. The article argues for the importance of robustness tests, which evaluate a model's ability to maintain consistent predictions under various input variations, as a more reliable indicator of generalization to real-world data. By using an example with the histopathology dataset Camelyon17-WILDS, where different hospital data introduces domain generalization challenges, the article demonstrates that models selected based on robustness tests, such as ResNet-101, can outperform those chosen solely on validation accuracy. Lakera has developed a tool, MLTest, to facilitate robustness testing, allowing users to assess model performance more effectively without new data, thus offering a better understanding of a model's potential in real-world applications.
Nov 16, 2025
1,454 words in the original blog post.
Lakera and Cohere have partnered to establish new security standards for Large Language Models (LLMs) amidst growing concerns about cybersecurity threats like prompt injection attacks, data leaks, and toxic language output. Recognizing that these issues affect not only LLM providers but also app developers and end-users, the two companies have collaborated to create resources such as the LLM Security Playbook and the Prompt Injection Attacks Cheatsheet. Their joint efforts include red-teaming activities to identify vulnerabilities in LLMs, allowing them to formulate strategies to mitigate potential risks. Both teams have participated in initiatives like DEFCON31's Generative Red Teaming AI Challenge, emphasizing the importance of community-wide collaboration to enhance the security of AI applications. Cohere, known for its enterprise AI platform, and Lakera, a leader in AI security solutions, are committed to refining security practices and staying abreast of emerging threats to ensure LLMs are deployed safely at scale.
Nov 16, 2025
839 words in the original blog post.
Lakera has announced the launch of Mosscap, a new AI security game at DEFCON, designed to tackle prompt injection vulnerabilities in Large Language Models (LLMs) such as OpenAI's ChatGPT, Anthropic's Claude, and Google's Bard. Building on the success of their previous game, Gandalf, Mosscap aims to provide an engaging and educational experience for participants, highlighting the critical nature of prompt injection vulnerabilities, which rank as the top threat on the OWASP Top 10 LLM Vulnerabilities Report. The game, inspired by the "Monk and Robot" series, offers players an opportunity to understand and mitigate these vulnerabilities, with data collected during gameplay being made available for research purposes. Founded by former engineers from major tech companies, Lakera is committed to enhancing AI security and education, with Mosscap being a key initiative in their mission to equip AI developers with essential security expertise. For more details, interested individuals can visit Mosscap's official website and follow Lakera on social media platforms.
Nov 16, 2025
562 words in the original blog post.
Large Language Models (LLMs) like GPT-4 have become crucial for various industries, enabling companies to enhance applications through models such as ChatGPT, Claude, and Cohere. The practice of fine-tuning foundation models on specific datasets has gained traction, allowing businesses to tailor pre-trained models for specific tasks, thus contributing to the rise of Generative AI. This process, which leverages the existing capabilities of models like BERT or GPT-4, requires understanding fine-tuning methods, applications, and challenges. The guide discusses how to choose the appropriate pre-trained model for fine-tuning, emphasizing the importance of security measures to protect LLMs from potential threats. Moreover, it highlights the iterative nature of fine-tuning, the need for domain-specific data, and the risk of issues such as overfitting and bias amplification. Tools like Lakera are suggested for safeguarding applications, and the guide provides insights into various fine-tuning strategies and resources, underscoring the importance of efficient techniques and security in deploying LLMs effectively.
Nov 16, 2025
4,066 words in the original blog post.
Lakera, a company involved in AI security, is joining forces with Check Point, as announced recently. David Haber, the CEO of Lakera, participated in a podcast episode of Safe Mode with Elias Groll from CyberScoop, discussing various aspects of AI security, including spearphishing and data protection challenges and solutions. Additionally, Lakera has contributed to accelerating Dropbox's journey in generative AI (GenAI) by providing expert-recommended security policies that can be implemented quickly to protect GenAI applications. The company offers resources, such as guides, to help secure these applications and provides updates on AI developments through newsletters.
Nov 16, 2025
297 words in the original blog post.
Lakera is partnering with Check Point to bolster the security of GenAI applications against malicious prompts, with a focus on offering expert-recommended security policies that can be easily applied and fine-tuned. The collaboration aims to enhance the security journey for companies like Dropbox by providing immediate protection and guidance for securing GenAI applications. Lakera's team emphasizes the importance of understanding and mapping large language model (LLM) usage and risk zones before scaling GenAI. Additionally, Lakera has launched a hacking simulation game called Gandalf: Agent Breaker and encourages individuals to subscribe to their newsletter for updates on their products and developments in the AI and LLM sectors.
Nov 14, 2025
229 words in the original blog post.
Daniel Graf has been appointed as the President of Lakera, bringing over 25 years of experience in scaling innovative technologies. Previously serving as the Chief Product and Technology Officer at DispatchHealth and having significant roles at Uber and Google, Graf has a proven track record in transforming industries, such as healthcare delivery and transportation, and developing global consumer technology products like Google Maps. At Lakera, he is expected to drive company strategy and expansion, focusing on securing AI applications against cyber risks, which aligns with Lakera's mission to support the development of AI technologies. The company's founder, David Haber, expressed confidence that Graf's expertise in leading hypergrowth companies will propel Lakera into its next growth phase, while Graf himself is eager to leverage his experience to further Lakera's mission during a transformative time for AI.
Nov 14, 2025
520 words in the original blog post.
Lakera has been recognized by the Financial Times as a "Tech Champion 2023" in the IT & Cyber Security category, highlighting its expertise in creating secure artificial intelligence applications. This accolade emphasizes the growing importance of AI in global technology, with Lakera being noted for its innovative solutions to AI vulnerabilities. The selection process involved nominations from FT readers and a thorough review by journalists and judges, culminating in the acknowledgment of Lakera's role in safeguarding AI's future trajectory. David Haber, CEO of Lakera, expressed pride in the recognition, viewing it as a reflection of the company's commitment to secure AI. The company encourages the AI community to explore the full list of Tech Champions, which celebrates the innovators and pioneers shaping the modern technological landscape.
Nov 14, 2025
540 words in the original blog post.
The Lakera AI Model Risk Index is an innovative security benchmark designed to test the resilience of large language models (LLMs) under real-world adversarial conditions, offering a more practical assessment of model security than traditional methods. By simulating various attack scenarios, such as direct prompt injections and indirect manipulations in real enterprise applications, the Index evaluates how effectively LLMs maintain their intended behavior under pressure. It provides a standardized 0–100 risk assessment score, enabling security teams to make informed decisions about model selection, deployment strategies, and governance, thereby moving from theoretical assumptions to actionable risk insights. This approach helps enterprises understand how models respond to adversarial inputs, ensuring they enforce behavioral boundaries while supporting safer GenAI deployment and compliance efforts.
Nov 14, 2025
1,640 words in the original blog post.
Lakera has introduced a Chrome extension called Privacy Guard, designed to protect users' private data during interactions with ChatGPT by using pattern recognition to detect potential data leaks. The extension operates locally on the user's machine, ensuring no data is stored or leaves the device, and offers notifications when sensitive information such as credit card numbers, email addresses, or social security numbers is detected. Users can choose to edit their prompts or proceed with submission, and can toggle specific detectors on or off based on their needs. While currently available for individual users on the Chrome Web Store, Lakera plans to expand the extension's compatibility to other Large Language Models and browsers, as well as offer enterprise solutions. The tool aims to enhance security without impeding the user experience, acknowledging the ongoing security challenges posed by advancements in Large Language Models.
Nov 14, 2025
915 words in the original blog post.
Lakera is set to participate in Black Hat USA 2025 in Las Vegas, where the team will engage with security leaders and AI innovators from August 2–7. The company is focusing on AI security, addressing the growing need for safeguarding GenAI systems as their adoption increases. Lakera offers solutions like Lakera Guard for real-time security of enterprise GenAI applications and Lakera Red for comprehensive red teaming to identify vulnerabilities. Attendees of Black Hat can book meetings with Lakera's team to explore how they are redefining AI security and learn about the latest attack vectors and mitigation strategies. Additionally, Lakera highlights its role in accelerating Dropbox's GenAI efforts and offers expert-recommended policies for securing GenAI applications. The company encourages interested parties to subscribe to their newsletter for updates on their products and developments in the AI LLM space.
Nov 14, 2025
391 words in the original blog post.
Lakera, a company dedicated to protecting enterprises from vulnerabilities in large language models (LLMs), has raised $20 million and announced its integration with Check Point. The team at Lakera has played a significant role in advancing Dropbox's journey in generative artificial intelligence (GenAI) by offering expert-recommended security policies designed to secure GenAI applications. These policies can be quickly applied and adjusted for enhanced protection from the outset. Additionally, Lakera has launched a hacking simulation game called "Gandalf: Agent Breaker" and provides resources to help organizations map their LLM usage and risk zones before scaling their GenAI initiatives. Users are encouraged to subscribe to Lakera's newsletter for updates on their products and news in the AI and LLM sectors.
Nov 14, 2025
228 words in the original blog post.
Lakera has been recognized as a Representative Vendor in Gartner's AI Trust, Risk and Security Management (AI TRiSM) Report, highlighting the company's dedication to addressing the unique security challenges posed by the rapid adoption of Generative AI (GenAI). Traditional security tools are often insufficient for the adversarial threats and data leakage risks associated with AI, necessitating a new approach to AI security. Lakera's AI Security Platform is designed to provide enterprises with the ability to securely integrate AI technologies while ensuring compliance with AI regulations and protecting confidential data. This recognition underscores the urgent need for specialized AI security solutions as AI continues to transform business operations, with Lakera positioning itself as a leader in offering scalable and effective security measures.
Nov 14, 2025
499 words in the original blog post.
Lakera, a leading real-time Generative AI (GenAI) security company, has released the 2024 GenAI Security Readiness Report, highlighting a significant gap in security preparedness amidst the rapid adoption of GenAI technologies. Despite 90% of surveyed cybersecurity experts actively using or considering GenAI, only 5% express confidence in their security measures against GenAI-specific threats, such as prompt attacks that can lead to unauthorized data access and system manipulation. The report, based on a survey of 1,000 cybersecurity professionals, also reveals concerns over LLM reliability, accuracy, and data privacy, with 22% having adopted AI-specific threat modeling. Lakera's educational AI game, Gandalf, which has attracted over a million players, including cybersecurity experts, underscores the ease with which GenAI systems can be manipulated by showcasing successful breaches by 200,000 players. Lakera, which recently secured $20 million in Series A funding from investors like Atomico and Citi Ventures, is poised to address these challenges by providing real-time, evolving AI defenses, and aims to track security preparedness annually.
Nov 14, 2025
900 words in the original blog post.
The blog post discusses the importance of red teaming in understanding and securing AI systems based on Large Language Models (LLMs), highlighting that these models introduce new security challenges distinct from traditional software vulnerabilities. It explains how LLMs, unlike conventional systems, can be manipulated through data inputs, turning them into attack vectors without direct system access. The article provides examples, such as adversarial SEO attacks and LLM-targeted exploits, to illustrate these vulnerabilities. It emphasizes the need for an advanced automated red teaming agent that surpasses human capabilities in identifying and exploiting weaknesses in AI applications, aiming to enhance security and trust in AI systems. The series aims to explore these challenges, define new vulnerabilities, and develop benchmarks to assess red teaming effectiveness, while acknowledging that traditional cybersecurity methods remain relevant but insufficient for the unique threats posed by LLMs.
Nov 14, 2025
1,476 words in the original blog post.
Lakera has been recognized as a leader in AI security, specifically in securing large language models, an accolade awarded by Sifted.eu and acknowledged by investors like Balderton Capital. The company has made significant advancements with its product, Lakera Guard, which effectively addresses cybersecurity threats such as prompt injections, data loss, and harmful content. Beyond its products, Lakera is committed to educating the industry on AI security threats, as exemplified by its educational game, Gandalf, which focuses on prompt injection attacks. This recognition highlights Lakera's pivotal role in advancing AI security solutions, marking a significant milestone in its mission to create a safer digital ecosystem.
Nov 14, 2025
408 words in the original blog post.
Large Language Models (LLMs) have advanced significantly in text generation and integration with external applications, but they also present security challenges that require careful management. The potential misuse by malicious actors raises concerns about social engineering, data exfiltration, and other security risks, prompting the need for comprehensive protective measures. LLM security involves safeguarding data, models, and infrastructure from unauthorized access and bias, with strategies ranging from fine-tuning models to implement ethical guidelines, to employing external censorship mechanisms. The implementation of security practices, such as regular audits and incident response planning, is critical for ensuring the reliability and trustworthiness of LLM outputs. Tools like Lakera Guard offer model-agnostic security enhancements for LLM applications, and organizations are encouraged to adopt best practices and comply with emerging regulations like the EU AI Act. The significance of governance, legal frameworks, and real-world insights, as well as resources like the AI Incident Database, are emphasized for a holistic approach to navigating the evolving landscape of LLM security.
Nov 14, 2025
6,407 words in the original blog post.
As AI systems, particularly Generative AI (GenAI), become more sophisticated, traditional cybersecurity measures struggle to keep pace, prompting innovative strategies like AI red teaming, which involves simulating real-world adversarial attacks to identify vulnerabilities in these dynamic, non-deterministic environments. Lakera's approach to AI red teaming highlights the need for adaptive and proactive defenses, emphasizing the unpredictable attack surface created by GenAI's reliance on natural language inputs and multimodal data, which encompass text, images, audio, and video. The company leverages insights from its threat intelligence platform, Gandalf, which harnesses crowd-sourced data to reveal vulnerabilities and develop cutting-edge security strategies, maintaining a balance between robust defenses and system usability. As GenAI systems continue to evolve towards more autonomous and agentic functionalities, the challenges in securing them will increase, necessitating continuous adaptation and innovation in red teaming methodologies to anticipate and mitigate emerging threats effectively.
Nov 14, 2025
2,512 words in the original blog post.
Lakera, a Zurich-based AI startup, won the 2021 DEKRA Award in the newly introduced "Startups" category, recognized for its innovative work on AI transparency, safety, and trustworthiness. The DEKRA Award, organized in collaboration with Germany's WirtschaftsWoche magazine, honors excellence in safety across various domains and included a virtual pitch competition among three finalists. Lakera's validation platform addresses the "black box" problem in AI by systematically testing AI systems to identify operating conditions, failure points, and risk assessments, promoting transparency. This platform is applicable across multiple industries, such as robotics, medical technology, Industry 4.0, and the automotive sector, and represents a significant step in ensuring AI safety and reliability.
Nov 14, 2025
503 words in the original blog post.
AI agents are autonomous systems designed to perform specific tasks by perceiving their environment, processing data, and making independent decisions, thereby distinguishing them from traditional software. These agents are composed of various modules, including perception, reasoning, action, learning, and communication, and come in different types such as simple reflex, model-based reflex, goal-based, utility-based, and learning agents. Their real-world applications span numerous industries, from customer service and autonomous vehicles to virtual assistants and gaming, offering benefits like task automation, increased efficiency, improved decision-making, reduced costs, and competitive advantage. However, AI agents also present challenges, including multiagent dependencies that may cause widespread failures, infinite feedback loops requiring human oversight, changing human-AI interactions, and evolving cybersecurity risks that necessitate updated protection strategies. As AI agents continue to impact the way we work and interact with technology, balancing their benefits with security considerations remains critical.
Nov 14, 2025
1,733 words in the original blog post.
Lakera has announced significant upgrades to its Lakera Guard's content moderation capabilities to enhance the security of generative AI (GenAI) applications against threats such as prompt injections and PII disclosure. The latest version of Lakera Guard boasts improved accuracy, nearly doubling the effectiveness of moderation across various categories, and a substantial reduction in latency, facilitating faster content analysis and real-time moderation. New features include a profanity detection system and broadened scope for detecting hate speech and sexually explicit content, ensuring comprehensive protection against offensive or harmful material. These enhancements extend Lakera Guard's enterprise-grade capabilities, providing high accuracy and low latency for GenAI applications in production, making it available to both SaaS and self-hosted customers with detailed documentation and support for seamless transition.
Nov 14, 2025
573 words in the original blog post.
Lakera has gained recognition from Help Net Security as one of the top cybersecurity startups to watch in 2024, reflecting its commitment to enhancing AI application security. The company collaborates with Fortune 500 firms, startups, and AI model providers, emphasizing the increasing importance of AI security. Lakera's flagship product, Lakera Guard, aims to protect AI systems from a variety of threats such as data breaches and prompt injections. Additionally, their initiative, Gandalf, is a prompt-injection game that educates users about AI threats and has been played by millions. In October 2023, Lakera secured $10 million in funding led by Redalpine, with participation from Inovia Capital, Fly Ventures, and industry experts. This recognition and funding support underscore Lakera's ongoing efforts to advance the field of AI security.
Nov 14, 2025
466 words in the original blog post.
The latest release of Claude Sonnet 4 presents significant advancements in large language model (LLM) security, particularly in its robustness against real-world adversarial attacks, setting a new standard compared to its predecessors and competitors like LLaMA 4 Maverick and ChatGPT 4.1, which showed vulnerabilities in various attack scenarios. Despite its improvements, Sonnet 4, like other LLMs, still faces challenges in dealing with complex adversarial prompts and requires additional security measures such as vulnerability scanning and guardrails for comprehensive protection. Anthropic's constitutional classifiers, integral to Claude's architecture, aim to mitigate harmful outputs by embedding ethical principles into model behavior, though they might encounter limitations in intricate real-world situations. The emphasis on security as a competitive advantage is underscored, highlighting the necessity for enterprises to integrate multi-layered defenses alongside deploying robust models like Sonnet 4 to ensure operational safety and reliability in generative AI applications.
Nov 14, 2025
1,248 words in the original blog post.
Developers have significant potential to address model and data bias in computer vision systems, with a key focus on ensuring data representativity during data collection and annotation. This process involves matching the collected data to the intended target demographic, such as ensuring a radiology diagnostic tool is tested on local demographics and machines used in the target hospitals. Collecting comprehensive metadata, like age and machine model, enhances the evaluation of machine learning models by identifying biases and ensuring all relevant data slices are present. It's crucial to test models on all demographic slices, including the less common ones, to avoid misleading aggregate metrics like accuracy and ensure the model performs well across the entire target population. The process emphasizes the importance of understanding target groups and maintaining data representativity to build reliable and unbiased computer vision systems.
Nov 14, 2025
840 words in the original blog post.
Reinforcement Learning from Human Feedback (RLHF) is an advanced machine learning technique designed to align artificial intelligence (AI) systems more closely with human values by incorporating direct human feedback. This approach addresses the limitations of traditional reinforcement learning, which often struggles with predefined reward systems that lack the ability to capture complex human preferences and ethical considerations. RLHF involves a comprehensive workflow that includes data collection from human feedback, supervised fine-tuning, reward model training, policy optimization, and iterative refinement, enabling AI to perform tasks that resonate with human intuition. Despite its potential to create AI models that are technologically sophisticated, ethically aligned, and socially beneficial, RLHF faces challenges such as scalability, cost, bias, and technical complexities in reward modeling and policy optimization. Recent advancements and alternative methods like Direct Preference Optimization (DPO) aim to mitigate these challenges, offering pathways to more efficient and effective AI systems. As RLHF continues to evolve, it holds promise in enhancing AI's applicability across various domains, fostering a future where AI systems are not only advanced but also ethically responsible and aligned with human values.
Nov 14, 2025
5,584 words in the original blog post.
Artificial intelligence (AI) has rapidly integrated into daily life, transforming industries and interactions with technology while also presenting significant risks that require diligent management. These risks include algorithmic bias, misinformation, job displacement, privacy violations, and potential misuse of AI in autonomous weapons and surveillance, which could exacerbate socioeconomic inequalities. The document emphasizes the necessity of robust AI risk management strategies and ethical frameworks to address these challenges, highlighting the importance of transparency, accountability, and human oversight in AI deployment. It discusses the specific vulnerabilities posed by Large Language Models (LLMs), such as prompt injection and training data poisoning, while suggesting mitigation strategies like comprehensive security solutions and adherence to frameworks like OWASP Top 10. The text advocates for international cooperation and regulatory measures, exemplified by the EU AI Act and USA's decentralized approach, to ensure that AI technologies benefit society while minimizing potential harms.
Nov 14, 2025
8,245 words in the original blog post.
Lakera has developed and released canica, a text dataset viewer designed to enhance the quality assessment of datasets used for training machine learning models. Canica allows users to interactively explore datasets as 2D plots using algorithms like t-SNE or UMAP, facilitating a visual understanding of data clusters and semantic relationships. The tool addresses challenges in dimensionality reduction by providing features that link the 2D visualizations back to the original embedding space, enabling users to explore local neighborhoods and focus on specific data subsets. Released under the MIT license, canica is available on GitHub and can be installed via pip, offering the machine learning community a new resource for data analysis and visualization, with further exploration and contributions encouraged through a tutorial notebook on GitHub.
Nov 14, 2025
918 words in the original blog post.
Agentic AI systems are evolving rapidly, presenting new security challenges as their capabilities expand beyond traditional conversational search into browsing, automation, and tool orchestration. This evolution, highlighted in Lakera's research, underscores the risks associated with over-privileged tools and uncontrolled browsing, where agents can inadvertently execute malicious code or republish harmful content due to implicit trust. The report emphasizes the importance of runtime guardrails, such as Lakera Guard and Lakera Red, to maintain security by enabling continuous monitoring and red-teaming to detect and mitigate vulnerabilities. These tools help organizations implement least privilege principles and validate content, ensuring that agents operate safely without compromising autonomy. The insights reveal that while agentic AI's speed and integration capabilities are impressive, they necessitate robust security measures to prevent misuse and ensure responsible innovation.
Nov 14, 2025
2,604 words in the original blog post.
Stress-testing is essential for ensuring the robustness and transparency of computer vision models, particularly when faced with real-world variables such as image quality degradation and occlusion. As models are developed for production, developers must address critical questions about the system's performance under varying conditions, which stress-testing can help answer by identifying the model's breaking points. This process involves using metamorphic relations and techniques from fuzz testing to explore how visual changes impact image annotations, aiding in both strategizing data collection and communicating the model's limitations to users. Lakera's MLTest offers advanced stress-testing capabilities that help developers pinpoint these vulnerabilities, allowing for improved model robustness and clearer user communication.
Nov 14, 2025
709 words in the original blog post.
Lakera has played a pivotal role in enhancing Dropbox's journey in implementing Generative AI (GenAI) by using Lakera Guard to secure their language models (LLMs). The collaboration offers Dropbox expert-recommended security policies that can be applied swiftly and adjusted as needed, providing immediate protection. This partnership underscores Lakera's commitment to helping organizations safeguard their GenAI applications effectively.
Nov 14, 2025
228 words in the original blog post.
Artificial intelligence security is crucial as AI systems, particularly large language models (LLMs) and generative AI, are vulnerable to manipulation, misuse, and attack throughout their lifecycle, from data collection to deployment and real-time interaction with users. Unlike traditional software, AI systems are dynamic and unpredictable, posing unique challenges that require new security approaches. Common threats include prompt injection, data leakage, and model theft, which exploit the AI's reasoning rather than its code. Effective AI security involves adaptive guardrails, threat-aware monitoring, and red teaming to mitigate these risks across the AI lifecycle. As AI becomes integral to critical operations, ensuring their security is not just a technical issue but a strategic imperative, demanding new strategies and continuous vigilance. Frameworks like OWASP, NIST AI RMF, and MITRE ATLAS provide guidance for securing AI systems. Looking ahead, AI security will evolve to address networks of autonomous agents, necessitating adaptive defenses akin to immune systems to keep pace with emerging threats.
Nov 14, 2025
4,258 words in the original blog post.
Computer vision models often suffer from "shortcut learning," where they rely on incorrect visual features for making predictions, which can lead to biases and poor generalization to unfamiliar environments. This issue arises due to the black-box nature of these models, making it difficult to detect and rectify the shortcuts within the learned features. Research has shown that models may perform well even when essential visual cues are masked, indicating reliance on irrelevant background features. This can be problematic, as demonstrated by a cow classification model that struggled to identify cows in atypical environments like beaches. To mitigate shortcut learning, examining metadata for statistical dependencies and conducting ablation experiments can help identify potential biases. Developing practices to detect these shortcuts during model evaluation is crucial for ensuring high-performing and reliable computer vision systems.
Nov 14, 2025
743 words in the original blog post.
Bias in machine learning (ML) systems is a critical issue that has recently garnered significant attention, as highlighted by the documentary "Coded Bias," which illustrates how algorithmic decision-making can lead to biased outcomes affecting entire populations. The inherent bias in data, stemming from societal, collection, and annotation factors, poses a challenge for those developing ML systems, who must ensure these systems do not perpetuate unfairness. Despite the difficulty in defining what constitutes a bias-free system, the ML community is encouraged to integrate rigorous testing processes, similar to those used in safety-critical systems, to mitigate bias. The discussion around algorithmic fairness should involve legal and regulatory experts to create concrete guidance, with notions such as "Demographic Parity" and "Equality of Opportunity" serving as starting points, despite their inability to be simultaneously satisfied. Recent regulatory proposals, like the EU's initiative to categorize "high-risk" AI systems, represent a step towards more structured and accountable development practices, emphasizing the need for thorough testing to ensure fairness across diverse demographics and scenarios.
Nov 14, 2025
1,130 words in the original blog post.
Developing computer vision (CV) products involves an exciting initial phase where early demos showcase the potential of machine learning models, but the real challenge lies in transitioning from proof-of-concept (POC) to production. While initial successes might lead to overconfidence, a rigorous assessment of machine learning models, including testing for robustness, bias, and data quality, is essential to avoid common pitfalls such as over-promising capabilities, resource wastage, and collecting inadequate data. Many AI projects falter in this transition, with 53% never reaching production, often due to insufficient attention to model robustness, inadequate data quality, and a lack of understanding among stakeholders about the model's limitations. Implementing comprehensive testing protocols early in the development process can mitigate these issues, ensuring that data and model performance align with specific use cases. Tools like Lakera's MLTest facilitate these assessments by providing automated insights into model performance and data quality, ultimately paving the way for more successful deployments.
Nov 14, 2025
1,165 words in the original blog post.
OWASP's Agentic AI Vulnerability Scoring System (AIVSS) has been developed to address the inadequacies of the Common Vulnerability Scoring System (CVSS) in evaluating risks associated with AI agents. While CVSS serves as a foundational tool for assessing vulnerability severity, it struggles with the dynamic and unpredictable nature of agentic systems where AI agents can act, improvise, and potentially conspire. AIVSS enhances CVSS by introducing an Agentic AI Risk Score (AARS), which factors in the amplification effects of AI autonomy, memory, and multi-agent interactions, and a Threat Multiplier to adjust scores based on active exploits. This enhanced system maintains the familiar 0-10 scoring range but provides a more accurate reflection of risks in AI contexts, enabling security teams to better prioritize vulnerabilities. Lakera, involved in shaping AIVSS, emphasizes its practicality in real-world scenarios, bridging the gap between traditional security measures and the evolving challenges posed by AI technologies.
Nov 14, 2025
1,300 words in the original blog post.
Generative AI-powered chatbots, utilizing Large Language Models (LLMs), are transforming technology interactions but also introducing significant security vulnerabilities such as data leakage, prompt injection, phishing, malware, and misinformation. These chatbots, while beneficial, require robust safeguards to protect sensitive data and maintain user trust. Key security measures include encryption, authentication, and authorization, alongside regular security audits and penetration testing. Organizations are urged to educate users about potential threats and maintain compliance with data protection regulations like GDPR and HIPAA, alongside adhering to AI ethics principles. Advanced security solutions, such as behavioral analytics, help detect unusual activities, while user education plays a crucial role in preventing phishing and other scams. By prioritizing security alongside the capabilities of LLM-powered chatbots, organizations can foster trusted and reliable AI interactions.
Nov 14, 2025
2,282 words in the original blog post.
Content moderation is evolving with the rise of generative AI (GenAI), which shifts the focus from post-publication policing of user-generated content to real-time moderation of AI-generated content. Traditional methods relying on keywords and human review are inadequate for the nuanced and multilingual outputs of large language models (LLMs), which can produce unpredictable and potentially harmful content. This new paradigm requires proactive interception of inappropriate or biased material at the point of generation, using advanced tools like Lakera Guard, which provides real-time, policy-driven moderation specifically designed for LLMs. This approach not only prevents risks such as prompt injections and evasive phrasing but also supports compliance with content standards, ensuring the safety and reliability of AI-powered applications. As companies like Dropbox integrate these solutions, the shift toward embedding moderation within the generation layer itself is becoming essential for maintaining trust and accelerating innovation in AI products.
Nov 14, 2025
2,240 words in the original blog post.
Lakera has been featured in a NIST report titled "Adversarial Machine Learning - A Taxonomy and Terminology of Attacks and Mitigations," which examines adversarial machine learning with a focus on classifying attacks and mitigation strategies. The report highlights Lakera’s effective techniques for defending large language models (LLMs) against prompt injection attacks, underscoring the importance of robust AI security measures. This recognition by NIST validates Lakera's work in AI security and serves as a vital resource for professionals aiming to secure AI systems against various adversarial challenges. Additionally, Lakera's team has played a role in enhancing Dropbox's GenAI initiatives and offers expert-recommended policies for securing GenAI applications, enabling users to implement protections efficiently.
Nov 14, 2025
359 words in the original blog post.
Metamorphic relations are presented as an effective method for enhancing machine learning model testing by expanding test coverage beyond what is achievable through standard data collection. These relations involve altering existing data in ways that maintain the original label, such as rotating images or adjusting color intensity, to address the test oracle problem where determining the correct output for a given input is challenging due to data scarcity and annotation costs. By applying metamorphic relations, such as image augmentations and temporal relations, models can be more robustly tested, ensuring they behave according to specifications even when subjected to various transformations. This approach not only multiplies available test data but also highlights that training with data augmentations does not guarantee model robustness, making testing essential for identifying potential bugs and enhancing model reliability. The concept has been successfully applied in various domains, including medical imaging and autonomous driving, revealing significant insights into model behavior and potential errors.
Nov 14, 2025
1,069 words in the original blog post.
Lakera is making strides in the AI security landscape, notably by joining forces with Check Point, and its CEO recently participated in the Datadog Cloud Security Lounge Podcast to discuss LLM security. The Lakera team has been instrumental in advancing Dropbox's GenAI initiatives by offering expert-recommended security policies for GenAI applications that can be applied quickly and adjusted as needed to provide immediate protection. Additionally, Lakera has launched a hacking simulation game called "Gandalf: Agent Breaker" and offers insights on mapping LLM usage and identifying risk zones before scaling GenAI efforts. Interested individuals are encouraged to subscribe to Lakera's newsletter for updates on their products and developments in the AI and LLM sectors.
Nov 14, 2025
235 words in the original blog post.
Social engineering, a manipulation technique exploiting human psychology to breach security systems, is evolving with the integration of artificial intelligence (AI), creating more sophisticated threats and opportunities for defense. Traditionally reliant on exploiting trust and human error, social engineering now utilizes AI to craft convincing phishing emails, conduct detailed reconnaissance, and simulate human interactions, enhancing the effectiveness of attacks. Conversely, AI also offers robust defensive capabilities, enabling the development of advanced detection algorithms and predictive analytics to identify and counter potential threats. The dual role of AI in social engineering—both as a tool for attackers and a defense mechanism—underscores the need for comprehensive strategies in cybersecurity, including employee education, AI-driven security tools, and robust authentication protocols. As AI continues to advance, its impact on social engineering tactics will likely increase, necessitating proactive and adaptive cybersecurity measures to mitigate the risks of AI-enhanced attacks.
Nov 14, 2025
4,838 words in the original blog post.
The European Union's proposed AI Act aims to regulate artificial intelligence by balancing innovation with safety, focusing on classifying AI systems based on their risk levels. It categorizes AI into unacceptable, high, and limited risk, imposing stringent compliance measures on high-risk systems to safeguard public safety and fundamental rights. Generative AI is not separately categorized but falls under existing categories with transparency obligations. Non-compliance with the Act could result in severe financial penalties, urging businesses to align with EU standards, necessitating a comprehensive review and enhancement of AI strategies and processes to ensure compliance. As the law is expected to be enforced within the next 12-24 months, companies are advised to prioritize transparency and prepare for the new regulatory landscape.
Nov 14, 2025
646 words in the original blog post.
Lakera has unveiled a new branding strategy to reflect its growth and commitment to AI security, as the company transitions from focusing solely on AI safety to addressing AI security challenges. Since its founding in 2021, Lakera has expanded from three co-founders to a global team, securing funding, and launching successful products like Gandalf, an AI education game with over a million players. The rebranding aims to communicate their dedication to supporting developers in creating secure AI products with the introduction of Lakera Guard, a product designed for applications using LLMs and multimodal foundation models. Collaborating with renowned design agency Koto, Lakera has updated its visual identity, including a new logo, color palette, and typography, to encapsulate its ethos of swiftly addressing AI threats. This new identity underscores their mission to empower developers and organizations to protect their GenAI applications from vulnerabilities, as Lakera continues to grow and recruit across various departments.
Nov 14, 2025
952 words in the original blog post.
Large Language Models (LLMs) are pivotal in the ongoing AI boom, significantly impacting industries reliant on language processing, such as healthcare, finance, and education. These models, like ChatGPT, have made AI more accessible to the public, showcasing near-human performance levels. LLMs function by predicting the next element in a text sequence through deep learning techniques, utilizing transformers and self-attention mechanisms. They are trained on vast datasets, often sourced from the internet, which presents challenges related to bias, privacy, and ethical considerations. Despite their impressive capabilities in generating text and applications in chatbots, code generation, and content creation, LLMs face limitations in logical reasoning and maintaining security. As foundational models, LLMs are typically fine-tuned for specific tasks, and their training demands substantial computational resources, often accessible only to elite companies. The article underscores the importance of understanding the technical and security aspects of deploying LLMs, while also addressing emerging research areas and potential risks associated with their use.
Nov 14, 2025
3,853 words in the original blog post.
Gandalf is a challenge created by Lakera to highlight the vulnerabilities of large language models (LLMs) and improve their defenses, particularly in contexts like healthcare and finance where data security is crucial. The game, stemming from an internal hackathon, involves trying to coax a language model into revealing a secret password, with each of the seven levels presenting increased difficulty as more sophisticated defenses are applied. As users progress, they encounter various strategies to prevent password leaks, such as checking both input and output for mentions of the password and employing additional language model checks. Despite these measures, users have found creative ways to bypass the defenses, demonstrating real-world implications for LLM security. Gandalf has gained significant popularity, registering millions of interactions and illustrating the ongoing challenge of securing AI applications against prompt attacks and other vulnerabilities.
Nov 14, 2025
2,759 words in the original blog post.
Computer vision models can encounter unexpected data during operation, leading to issues such as data drift, where models may fail to perform as expected with new input data. This can happen, for instance, when a hospital changes its x-ray machine but continues using the same diagnostic model, or when an autonomous vehicle trained on European roads is deployed in an American city. To address these issues, it is crucial to implement out-of-distribution detection systems that identify suspicious or unknown inputs and allow the system to fail gracefully, thereby involving human intervention when necessary. This approach helps mitigate operational bias and prevent silent failures by keeping the data and models up-to-date and ensuring that mitigation strategies are in place. Out-of-distribution detection is a significant component of many learning systems, such as Generative Adversarial Networks, which use a discriminator network to identify suspicious images. Maintaining awareness of data drift and updating models is essential for the ongoing lifecycle of AI systems.
Nov 14, 2025
604 words in the original blog post.
Lakera, an AI security company, is aligning with the MITRE ATLAS framework to proactively mitigate adversarial risks associated with AI systems. The MITRE ATLAS framework, developed by the not-for-profit organization MITRE, provides a comprehensive knowledge base of adversary tactics and techniques targeting AI systems, highlighting vulnerabilities as AI is increasingly integrated into various industries. Lakera employs solutions like Lakera Guard and Lakera Red, which are designed to monitor, detect, and respond to adversarial attacks on machine learning models and AI applications, particularly those powered by Large Language Models (LLMs). These solutions cover vulnerabilities such as prompt injection, phishing, insecure LLM plugins, and data poisoning, offering a robust security infrastructure. Lakera Guard focuses on real-time threat assessment and defense against prompt injections, while Lakera Red specializes in identifying and addressing LLM security vulnerabilities before AI applications are deployed. By leveraging a vast database of threat intelligence and continuous stress-testing, Lakera aims to safeguard AI systems against evolving security threats, ensuring their integrity and reliability.
Nov 14, 2025
1,986 words in the original blog post.
The webinar titled "Life vs. ImageNet" hosted by Lakera explored the complexities and challenges of transitioning machine learning (ML) and computer vision technologies from academic settings to real-world applications. Panelists from various industries, including autonomous driving, healthcare, and technology, shared insights into the significant differences between academic ML, which often begins with defined datasets and focuses on model performance, and real-world ML, which prioritizes product specifications and user needs. Key takeaways emphasized the importance of adopting a product-first mentality, understanding diverse evaluation metrics, and addressing challenges unique to scaling ML systems, such as data representation, robustness, and transparency. The discussion also highlighted the challenges faced by traditional industries and the rise of foundation models, which present new complexities due to their pre-trained nature and the potential biases in the data they use. The panelists underscored the necessity of robust evaluation frameworks and a focus on testing and validation to build trust and transparency in AI solutions.
Nov 14, 2025
1,920 words in the original blog post.
In the evolving landscape of data security, traditional Data Leakage Prevention (DLP) systems are increasingly inadequate in addressing the challenges posed by Generative AI (GenAI) technologies. Unlike static data, which could be controlled through regex and pattern matching, GenAI systems dynamically process data through summarization, translation, and reasoning, often exposing sensitive information in unexpected ways. These systems, including large language models (LLMs) and autonomous agents, can inadvertently leak data through paraphrasing or summarizing confidential information without breaching traditional security perimeters. As a result, a modern DLP strategy must be language-native and context-aware, capable of understanding language intent rather than relying solely on pattern matching. This approach involves real-time monitoring of interactions, memory access, and agent workflows, ensuring that security policies are dynamically applied based on user identity, context, and data usage. Companies like Lakera are pioneering this new paradigm by developing AI-native detectors and real-time monitoring systems that address the unique vulnerabilities introduced by GenAI, aiming to secure sensitive data while maintaining productivity and efficiency in AI-driven environments.
Nov 14, 2025
1,943 words in the original blog post.
The Lakera LLM Security Playbook provides a comprehensive resource for understanding AI security, focusing on the distinction between prompt attacks and non-prompt attacks in generative AI. It offers a detailed analysis of vulnerabilities in large language models (LLMs) and includes practical advice on data sanitization and personal identifiable information detection. The guide introduces Gandalf, an online game for learning AI security, and highlights the Lakera Guard security solution to counter AI threats. Featuring a database with nearly 30 million LLM attack data points, the playbook uses real-world examples to clarify common misconceptions and offers actionable guidelines for mitigating vulnerabilities, making it a valuable tool for professionals and enthusiasts aiming to secure AI systems effectively.
Nov 14, 2025
335 words in the original blog post.
Generative AI, known as GenAI, is a revolutionary segment of artificial intelligence focused on creating new content by learning patterns from input data, using methods like unsupervised and semi-supervised learning. Technologies such as large language models (LLMs) and Generative Adversarial Networks (GANs) have significantly advanced GenAI's ability to produce text, images, videos, and audio, transforming industries from digital art to healthcare. Recent innovations like Transformers have broadened GenAI's applications, enhancing tasks in natural language processing and image synthesis. Tools like ChatGPT, DALL·E, and BARD exemplify how GenAI is reshaping content creation, with benefits that include increased productivity and cost savings. However, the technology also poses risks, such as data privacy concerns and the potential misuse of deepfakes, necessitating careful management and ethical deployment. Looking ahead, GenAI is poised to become an integral part of enterprise applications, driving innovation and efficiency while requiring vigilant risk mitigation.
Nov 14, 2025
3,343 words in the original blog post.
Data poisoning has transitioned from an academic concept to a tangible threat in 2025, affecting various stages of the AI model lifecycle, including pre-training, fine-tuning, retrieval, and tools. This adversarial attack involves inserting corrupted or biased data into models' learning processes, leading to compromised outputs, hidden backdoors, or biased behavior. The severity of this threat is underscored by real-world incidents, such as backdoors in GitHub code and poisoned synthetic data pipelines, highlighting how even minimal contamination can have outsized impacts. Effective defenses require a comprehensive approach, combining data provenance, adversarial testing, and runtime guardrails to protect against both external and internal threats. Researchers and practitioners are urged to evolve benchmarks and defenses rapidly to address the complex challenges posed by data poisoning, ensuring AI systems remain reliable and trustworthy in critical applications.
Nov 14, 2025
3,108 words in the original blog post.
Lakera has been recognized in Gartner's "Emerging Tech: Top 4 Security Risks of GenAI" report, which addresses the significant security challenges posed by generative AI technologies, including issues related to privacy, data security, attack efficiency, misinformation, and identity fraud. The acknowledgment highlights Lakera's role in developing effective security measures to mitigate these threats, reflecting the company's capabilities in detecting and addressing GenAI-related risks. For professionals in AI security, the Gartner report provides crucial insights and strategies for managing these challenges, and Lakera's involvement underscores its contribution to enhancing the safe deployment of AI technologies across various sectors. Additionally, Lakera offers expert-recommended security policies for securing GenAI applications, enabling users to apply and customize them easily for immediate protection.
Nov 14, 2025
365 words in the original blog post.
Effective machine learning systems hinge on high-quality data, making data bugs a significant concern that engineers must address during development. These bugs can arise from issues like missing values, incorrect annotations, data inconsistencies, and corrupted or duplicated data, which can skew model training and evaluation. Conversely, having the right data is equally crucial, as a lack of representative data can lead to inadequate system performance, such as an autonomous driving model failing on roundabouts if such scenarios are underrepresented in the training data. To combat these challenges, implementing robust data tests is essential, ensuring that the datasets are both correct and comprehensive from the outset. Mature teams prioritize data quality testing early in their projects to prevent unnoticed data bugs from causing delays, emphasizing continual monitoring to maintain data integrity.
Nov 14, 2025
1,197 words in the original blog post.
Large Language Models (LLMs) like OpenAI's GPT-3 and GPT-4 have transformed technology interactions but pose significant cybersecurity challenges. To address these issues, a variety of tools have been developed to secure LLM applications, focusing on mitigating risks such as unauthorized access, model exploitation, and data leakage. Among these tools are Lakera Guard, WhyLabs LLM Security, and Lasso Security, each offering unique features like prompt injection protection, data loss prevention, and threat modeling. These tools are part of a broader effort to enhance the security of LLMs by incorporating comprehensive assessments, monitoring capabilities, and proactive threat identification. As threats continue to evolve, the role of these tools in providing robust security measures becomes increasingly crucial, ensuring that LLMs can be deployed safely and effectively across various applications. The ongoing development and adaptation of security tools reflect the need for flexible and forward-looking solutions to safeguard against the expanding landscape of cybersecurity threats associated with LLMs.
Nov 14, 2025
3,984 words in the original blog post.
Foundation models have revolutionized machine learning by serving as versatile, large-scale AI architectures that can be fine-tuned for various applications without extensive retraining. These models, such as GPT-4, BERT, and DALL-E, have emerged due to advancements in deep learning and the availability of vast datasets, allowing them to be used in diverse fields ranging from natural language processing to image generation. Their adaptability and robustness make them foundational to the future of AI, although they come with challenges such as high computational costs, privacy concerns, and the need for domain-specific adaptation. Despite these hurdles, foundation models are expected to generate significant economic value and continue evolving towards more efficient, ethical, and secure applications, with collaborative efforts guiding their responsible development and deployment.
Nov 14, 2025
3,581 words in the original blog post.
Lakera's recent report highlights the rapid adoption of Generative AI (GenAI) technologies across industries, with nearly 90% of organizations actively implementing or planning to explore large language model (LLM) use cases. Despite this widespread adoption, only about 5% of organizations express high confidence in their AI security measures, indicating a significant gap between usage and preparedness. The report, which includes insights from professionals at prominent companies like Disney and Citibank, underscores the urgent need for AI-specific security frameworks that can adapt to evolving threats, such as prompt injection attacks and the jailbreaking of AI systems. Key concerns include the lack of understanding among engineers and security teams about the intricacies of LLMs, as well as the inadequacy of traditional security methods to address AI-related vulnerabilities. As AI technology advances, the report advocates for a paradigm shift towards developing AI-driven security solutions that evolve alongside the threats they are designed to counter.
Nov 14, 2025
1,085 words in the original blog post.
Lakera's research paper, "Gandalf the Red: Adaptive Security for LLMs," addresses the need for adaptive defenses in the rapidly evolving landscape of Large Language Model (LLM) security, emphasizing that static defenses are insufficient against increasingly sophisticated attackers. The study introduces the Dynamic Security and Utility Threat Model (D-SEC), which helps balance the trade-offs between security and usability, highlighting the importance of adaptive defenses that evolve alongside attacker strategies. Key strategies outlined include restricting application scope, implementing defense-in-depth, and leveraging adaptive security measures based on user behavior to enhance protection without significantly degrading usability. The research also showcases Gandalf, a gamified red-teaming platform that aids in uncovering LLM vulnerabilities by leveraging human creativity and interactive feedback, thus informing the development of more robust security strategies. The findings underscore the challenge of maintaining a balance between robust security measures and maintaining a positive user experience, particularly in the context of LLM applications.
Nov 14, 2025
1,426 words in the original blog post.
On February 22, 2024, Microsoft announced PyRIT, a new tool designed to enhance AI security by helping experts identify and mitigate risks in AI applications. Highlighted in this announcement was Lakera's game, Gandalf, which serves as an interactive educational tool to demonstrate AI security principles, showing how PyRIT can be applied to jailbreak the game. Gandalf, praised for its engaging approach to AI security education, has been played by millions and illustrates the increasing complexity and challenge of securing AI systems. Lakera, committed to advancing AI safety, offers resources like Lakera Guard to protect AI systems from threats such as data breaches and prompt injections, while continuing to support organizations in deploying secure AI technologies.
Nov 14, 2025
567 words in the original blog post.
Large language models (LLMs) have significantly evolved, now excelling in both contextual conversations and programming tasks, and their development has been heavily influenced by the open-source movement. This article examines the top 11 open-source LLMs, detailing their capabilities, challenges, and best practices for selecting the right model for specific needs. It highlights models like Llama 2, OpenLLaMA, and Falcon, which offer varying parameter sizes and applications, from dialogue optimization to multilingual text generation. The text also discusses the challenges of open-source LLM development, such as cost, privacy, bias, and scalability, while offering insight into model evaluation through leaderboards and the importance of security measures against vulnerabilities like prompt injection attacks. The open-source LLM landscape is poised for further innovation, driven by a commitment to ethical, user-centric models that balance the benefits of accessibility and customization with the complexities of development and deployment.
Nov 14, 2025
3,549 words in the original blog post.
Prompt engineering is a crucial skill for optimizing the performance, reliability, and safety of large language models (LLMs) such as GPT-4o, Claude 4, and Gemini 1.5 Pro, which, by 2025, have advanced significantly. Effective prompt engineering involves more than clever wording; it requires a deep understanding of formatting techniques, reasoning scaffolds, role assignments, and potential adversarial exploits to consistently improve model outputs. The practice not only enhances usability but also mitigates security risks by preventing adversarial prompt injections that could bypass LLM guardrails. With models responding differently to various formatting patterns, prompt engineering becomes a critical tool for aligning model behavior with human intent, controlling tone and structure, and ensuring safety. Beyond technical users, prompt engineering is accessible to non-coders like product managers and UX writers, making it a valuable skill across industries. As AI systems become more integrated into products, the ability to craft effective prompts will be as essential as writing clean code, emphasizing the importance of a structured approach to prompt design for achieving reliable and trustworthy AI interactions.
Nov 14, 2025
9,147 words in the original blog post.
Large language models (LLMs) are inherently multilingual, processing numerous languages, yet their security measures are often designed with an English-first approach, leaving them vulnerable to attacks in other languages. This gap allows adversaries to exploit non-English prompts, code-switching, and translation-based tricks to bypass AI safeguards, leading to potential data leaks and inconsistent policy enforcement. Evidence shows real-world cases where attackers have successfully bypassed AI defenses by using multilingual queries, revealing critical security risks. To mitigate these threats, it's crucial for businesses to implement multilingual security strategies that address these vulnerabilities and ensure consistent protection across all languages. Lakera Guard exemplifies a solution that offers comprehensive AI security for over 100 languages, emphasizing the need for global, multilingual security measures as AI becomes more integrated into business operations.
Nov 14, 2025
1,945 words in the original blog post.
The Backbone Breaker Benchmark (b3), developed by Lakera in collaboration with the UK AI Security Institute, is a novel approach to evaluating the security of AI agents by focusing on the vulnerabilities within their core large language models (LLMs). Unlike traditional benchmarks that assess the intelligence or safety of a model as a whole, b3 zooms in on the individual steps where LLMs may fail under targeted attacks, utilizing a method called threat snapshots. These snapshots isolate specific moments when an AI agent might make a vulnerable decision, allowing for a focused and reproducible evaluation of LLM security. The b3 benchmark employs nearly 200,000 human red-team attempts from the Gandalf: Agent Breaker project to create a comprehensive dataset for testing models against real-world adversarial scenarios. Findings reveal that models with explicit reasoning processes tend to be more secure and that open-weight models are rapidly closing the security gap with their closed-weight counterparts. The benchmark aims to transform AI security into a measurable and comparable science, offering valuable insights for developers, model providers, researchers, and policymakers, with the ultimate goal of establishing a new standard for evaluating AI agent security.
Nov 14, 2025
2,245 words in the original blog post.
Lakera, a Zurich-based startup founded in 2021, has been recognized as one of Switzerland's top 10 startups for 2024 by TechRound, highlighting its innovative efforts in AI security. The company, which comprises a diverse team from over 15 nationalities, aims to address the critical challenge of AI system and application security as they detect over 100,000 AI attacks daily. Lakera's popular AI security game, Gandalf, has been instrumental in providing insights into securing AI systems and has contributed significantly to their expertise in AI security intelligence. CEO David Haber emphasizes the company's commitment to advancing AI safety and empowering developers and organizations to build secure AI applications. As Lakera continues to shape the future of AI security, they invite interest in the broader ecosystem of innovative Swiss startups noted by TechRound.
Nov 14, 2025
514 words in the original blog post.
Understanding and effectively monitoring Large Language Models (LLMs) is crucial for modern data professionals due to the transformative potential and inherent challenges of these AI tools. Key aspects of LLM monitoring include ensuring accuracy, speed, context relevance, and fairness, while also addressing risks such as prompt injection, data disclosure, and excessive autonomy. Monitoring involves real-time evaluation of performance metrics like response time and sentiment analysis, while also considering operational aspects like latency and throughput. The importance of monitoring lies in preventing security breaches, building user trust, and improving system performance. Best practices include using specialized tools like Lakera Guard for real-time blocking, implementing data sanitization, and maintaining human oversight. As the field of LLMs evolves, continuous learning and adaptation are essential to ensure secure, reliable, and effective AI systems.
Nov 14, 2025
3,226 words in the original blog post.
Reinforcement Learning (RL) is a transformative approach within artificial intelligence, emphasizing a paradigm shift in machine learning where agents learn through interaction and trial-and-error rather than relying on pre-fed data. RL agents operate in environments by taking actions and receiving feedback in the form of rewards or penalties, enabling them to optimize strategies for achieving specific goals. This method is increasingly applied across diverse sectors, including gaming, autonomous vehicles, energy optimization, and healthcare, offering solutions with enhanced efficiency and minimal human intervention. RL's conceptual framework involves understanding core elements such as agents, environments, actions, states, and rewards, which together create a foundation for developing intelligent systems capable of addressing complex challenges. The exploration vs. exploitation dilemma presents a significant aspect of RL, requiring a balance between trying new actions and utilizing existing knowledge, a challenge navigated through strategies like epsilon-greedy, Upper Confidence Bound, and Thompson Sampling. RL's potential is further amplified by model-based and model-free approaches, with applications extending to AI security, robotics, finance, and personalized medicine, illustrating its role as a pivotal technology for future AI advancements.
Nov 14, 2025
5,054 words in the original blog post.
Lakera, a leading security platform for generative AI applications, has launched the AI Model Risk Index, a new standard for evaluating the security of large language models (LLMs) against attacks. This index is designed to measure real-world risk exposure and how effectively these models can maintain their intended behavior under adversarial conditions. It tests LLMs across various industries, including technology, finance, healthcare, law, and education, by simulating real-world attacks and assessing the models' ability to function predictably under threat. Unlike traditional cybersecurity frameworks, the AI Model Risk Index focuses on practical questions about model manipulation and mission-specific rule adherence. It provides quantitative risk measurements, allowing enterprises to compare different AI models' security and track changes over time. The report highlights that newer, more powerful LLM versions may not necessarily be more secure and that all models can potentially be manipulated. Lakera, founded by David Haber, Mateo Rojas-Carulla, and Matthias Kraft in 2021, operates from Zurich and San Francisco and continues to advance AI defenses through research and tools like their viral AI security game, Gandalf.
Nov 14, 2025
770 words in the original blog post.
Lakera, a company specializing in AI security, recently participated in DEFCON31, a prominent cybersecurity conference in Las Vegas, where they sponsored the AI Village and introduced Mosscap, a game designed to engage users in AI security challenges. The event highlighted the increasing focus on large language model (LLM) security, with enterprises prioritizing the prevention of prompt injections, data leakage, and model misuse. Lakera's involvement included the Generative Red Team (GRT) Challenge, where participants tackled tasks to exploit vulnerabilities in models from major AI providers. The conference underscored the importance of AI security and the need for regulations, as emphasized by experts like Craig Martell from the U.S. Defense Department and Nicolas Moës from The Future Society. DEFCON31 also marked the beta launch of Lakera Guard, an API aimed at enhancing LLM security, which received positive feedback from the AI security community and stimulated discussions with policymakers.
Nov 14, 2025
1,380 words in the original blog post.
Artificial intelligence (AI) risk management is essential as AI's capabilities expand, focusing on identifying and mitigating the unique risks associated with AI systems. This field is distinct from using AI for risk management in sectors like finance or healthcare. Key frameworks, including the NIST AI Risk Management Framework, ISO/IEC 23894:2023, the EU AI Act, and McKinsey's approach, offer structured guidelines for addressing technical, ethical, and societal challenges, ensuring responsible innovation. These frameworks emphasize adaptability, consistency, and collaboration among business, legal, and technical teams to manage risks throughout the AI lifecycle. The EU AI Act introduces a risk-based categorization system, influencing global AI practices, while Lakera provides specialized solutions for securing Large Language Model (LLM) based systems. Continuous adaptation and proactive risk management are crucial as AI technologies evolve, requiring organizations to constantly refine their practices to address emerging vulnerabilities and ethical concerns.
Nov 14, 2025
2,375 words in the original blog post.
Lakera Guard's Fall '25 release introduces innovative features to enhance AI security by adapting to applications, minimizing noise, and maintaining low latency, ensuring global AI experiences remain safe, accurate, and swift. The update includes adaptive calibration, which adjusts flagging criteria based on specific application data to reduce false positives and irrelevant alerts, achieving unprecedented accuracy rates. Additionally, the multilingual moderation model provides comprehensive content protection across over 100 languages, significantly reducing errors and improving detection accuracy, particularly in low-resource languages. The consistent latency architecture allows real-time interaction by running detectors in parallel and prioritizing intelligently, offering sub-40ms response times for short prompts and under 200ms for longer ones. These advancements are part of Lakera's broader mission to protect AI systems against evolving threats, ensuring security solutions keep pace with modern demands.
Nov 14, 2025
1,002 words in the original blog post.
During the World Economic Forum 2024, the AI House Davos panel titled "AI Safety Unplugged: Navigating the Risks Without the Hype" featured prominent figures like Yann LeCun from Meta, David Haber of Lakera, Seraphina Goldfarb-Tarrant from Cohere, and MIT's Max Tegmark discussing AI's challenges and future. Tegmark emphasized the dangers of AI-driven misinformation and deep fakes as elections approach, while Goldfarb-Tarrant highlighted the lack of robust evaluation methods in large-scale AI deployments. Haber introduced the concept of the "Internet of Agents," predicting increased cyber risks as AI systems become more interconnected. LeCun, skeptical about the imminent arrival of AGI, underscored the limitations of autoregressive language models, stressing the complexity of measuring intelligence. The panel advocated for a balanced approach to open-source AI development to avoid power concentration and urged a focus on immediate, certain AI risks rather than speculative future threats, emphasizing the need for clear understanding and control over AI capabilities by stakeholders.
Nov 14, 2025
898 words in the original blog post.
Prompt injections, a type of attack on language models, have become a significant security concern as businesses increasingly integrate large language models (LLMs) into their applications. These attacks can be classified into direct and indirect prompt injections, with the former allowing attackers to manipulate the input to an LLM directly. A specific form of direct prompt injection known as "jailbreaking" enables attackers to bypass model restrictions, potentially leading to unauthorized actions such as exfiltrating sensitive information or executing arbitrary commands. The article emphasizes the importance of developing defenses against prompt injections, highlighting strategies like privilege control, input and output sanitization, and human oversight. As organizations like OWASP work on standards for LLM vulnerabilities, companies are urged to swiftly implement protective measures to safeguard their systems.
Nov 14, 2025
1,349 words in the original blog post.
AI technology is increasingly being integrated into business operations, yet it brings significant security challenges that organizations must address. While 49% of firms are using tools like ChatGPT across various departments, the adoption of AI has led to heightened concerns about compliance, privacy, and ethics, particularly with generative AI. The paradox of AI in cybersecurity is evident, as 93% of security professionals believe AI can enhance cybersecurity, but 77% of organizations feel unprepared to counter AI threats. The AI cybersecurity market is rapidly expanding, forecasted to grow from $22.4 billion in 2023 to $134 billion by 2030, driven by the escalating sophistication and volume of cyber threats. Despite the potential benefits of AI in improving data security, 60% of respondents fear their organizations are ill-equipped to defend against AI-powered attacks. Regulatory efforts are underway, with the EU AI Act leading the way in establishing guidelines for safe and ethical AI development. As organizations continue to explore and implement AI technologies, it is crucial for them to develop comprehensive strategies to remain compliant with evolving regulations and effectively manage the associated risks.
Nov 14, 2025
2,515 words in the original blog post.
Machine learning (ML) development often lacks the rigorous testing and release processes of traditional software engineering, leading to systems that perform well in controlled environments but fail in real-world situations, posing risks such as undetected pedestrians or flawed COVID diagnostics. While traditional software benefits from test-driven development and continuous integration, ML systems frequently adopt a "ship-to-test" approach, resulting in vulnerabilities only emerging during operation. Lakera aims to address these issues by integrating systematic testing into ML development, with their MLTest tool automatically identifying vulnerabilities before deployment, thereby enhancing the reliability and safety of AI products. By adopting software engineering best practices, ML teams can improve the quality and speed of AI product development, mitigating risks associated with insufficient testing.
Nov 14, 2025
906 words in the original blog post.
Responsible AI is an emerging field focused on ensuring that artificial intelligence is developed and used ethically, transparently, and safely, incorporating principles such as accountability, privacy, fairness, and human-centered design. Industry leaders like Microsoft, Google, and global frameworks from institutions like NIST set standards and practices such as transparency in AI processes and accountability throughout the AI lifecycle. The challenges in implementing Responsible AI include data bias, the need for regulation, and technical limitations, with efforts being made to mitigate these through continuous testing, diverse team involvement, and global collaboration. Tools like Google's Responsible AI toolkit and Microsoft's Fairlearn are examples of resources aiding in these efforts. The global perspective on AI governance varies, with countries like China, the USA, and the EU adopting different approaches based on regional priorities, yet international collaborations like the OECD's AI principles aim to harmonize ethical standards. As AI technologies continue to evolve, the field of Responsible AI will be crucial in guiding their development to align with societal values and ethical norms.
Nov 14, 2025
3,351 words in the original blog post.
Regression testing is a crucial technique in machine learning that ensures models maintain consistent performance despite changes in data and parameters, by re-running tests to confirm that previously resolved bugs do not reappear. This method is particularly important as machine learning systems evolve and datasets constantly change, potentially leading to reoccurring bugs. A practical approach is to create a "difficult cases" dataset from inputs that cause model failures, using it as a regression test set to track and improve performance on known weak spots. The article highlights real-world applications like a computer vision system for Olympic events that struggles with shadows, suggesting the creation of targeted regression datasets to enhance model robustness under varying conditions. Tesla's large-scale regression testing for its autopilot system exemplifies proactive strategies, emphasizing the importance of mining edge cases and continuously evaluating system behavior. Even on a smaller scale, the principles of proactive regression testing can be applied to improve generalization and reliability, ultimately building more trustworthy machine learning models.
Nov 14, 2025
1,045 words in the original blog post.
Lakera has developed an implementation of OpenAI's CLIP model that eliminates the need for PyTorch, facilitating easier deployment on production and edge devices. CLIP, known for its image-to-text capabilities, typically relies on PyTorch for its three main components: the text tokenizer, the image preprocessor, and the model itself, which outputs cosine similarities of text and image embeddings. Lakera has rewritten the text tokenizer in NumPy, created a custom image preprocessor, and exported the CLIP model to an .onnx format to replace PyTorch with the more lightweight onnxruntime. This development allows for a more streamlined and accessible application of CLIP in various environments.
Nov 14, 2025
494 words in the original blog post.
Lakera has announced enhancements to its Lakera Guard platform, focusing on advanced Personally Identifiable Information (PII) Detection and Data Loss Prevention (DLP) for enterprise applications. These improvements are designed to secure GenAI applications and comply with regulations such as GDPR, GLBA, and HIPAA by offering robust solutions for identifying and managing PII. The platform's upgraded detection algorithms have increased accuracy and reduced false positives and latencies, surpassing many industry-standard and open-source solutions. Key features include the ability to detect a variety of PII entities like credit card numbers, email addresses, and social security numbers, with seamless API integration for easy application deployment. Lakera Guard's advanced capabilities are available to both SaaS and self-hosted customers, reinforced by comprehensive documentation and support for smooth integration and deployment, setting a high standard for enterprise-grade data protection.
Nov 14, 2025
705 words in the original blog post.
The advancement and widespread integration of Large Language Models (LLMs) such as OpenAI's ChatGPT, GPT-4, Claude, Google's Bard, Anthropic, and Llama have led to significant ethical and security concerns, particularly regarding the concept of "jailbreaking." This term, borrowed from the world of smartphones, refers to bypassing built-in safeguards of LLMs to manipulate them into producing harmful or inappropriate content using techniques such as adversarial prompts. These vulnerabilities are exploited through methods like prompt injection, prompt leaking, and roleplay jailbreaks, posing risks to data security and operational integrity across industries. As LLMs become more central to various applications, understanding these threats and implementing robust defenses—such as red teaming, AI hardening, and continuous security education—becomes crucial to safeguard their usage and maintain trust in AI systems.
Nov 14, 2025
3,414 words in the original blog post.
Prompt injection is a significant and evolving AI security threat, where attackers manipulate large language models (LLMs) by embedding deceptive instructions to override system prompts, extract sensitive data, and subvert AI-driven decision-making. Unlike traditional cybersecurity attacks that target code vulnerabilities, prompt injection exploits the model's instruction-following logic, using language to influence the system's behavior. This threat is identified as the top AI security risk by OWASP and poses challenges for enterprises deploying AI applications, particularly in sensitive domains like finance and healthcare. Despite advancements in LLMs, attackers continue to refine their methods, necessitating a proactive and multi-layered security approach that includes real-time detection, continuous adversarial testing, and adaptive defenses to safeguard AI systems.
Nov 14, 2025
4,181 words in the original blog post.
Gandalf, an AI security education platform created by Lakera, has unveiled a redesigned user interface aimed at enhancing usability and educational value. Initially developed during a hackathon to address vulnerabilities in large language models (LLMs), Gandalf has become a widely-used tool for understanding AI security, with over a million players engaging in its challenging prompt-injection game. The updated interface includes new features such as educational materials and a section called "Gandalf's AI Security Vault" to deepen users' understanding of AI vulnerabilities. Despite the changes, the core gameplay remains intact, offering progressively difficult levels that teach AI security concepts. Gandalf has gained traction among leading tech companies and educational institutions, highlighting its impact on the AI security landscape. The platform's success is further recognized through its inclusion in Microsoft's PyRIT toolkit and coverage in prominent media outlets. As Lakera continues to innovate, new Gandalf adventures are in development, promising fresh challenges and learning opportunities for users.
Nov 14, 2025
1,207 words in the original blog post.
Gandalf: Agent Breaker is a hacking simulator game developed by Lakera that challenges players to exploit real-world vulnerabilities in agentic AI applications, highlighting the security risks associated with GenAI systems. This interactive game allows users to engage with realistic scenarios, such as prompt attacks, memory tampering, and tool abuse, to understand how these systems can be broken and why traditional security measures may be inadequate. By navigating through multiple levels of increasing difficulty, players gain hands-on experience in identifying and mitigating these vulnerabilities, learning to think like a hacker and develop effective defenses. Gandalf: Agent Breaker is designed for a diverse audience, including security professionals, AI engineers, and gamers, and offers a global leaderboard for users to showcase their skills and strategies in securing GenAI applications.
Nov 14, 2025
1,289 words in the original blog post.
As GenAI applications grow more complex, the Model Context Protocol (MCP) has emerged as a critical standard for connecting AI models to real-world data and tools, but it also introduces security vulnerabilities that traditional defenses may not address. Lakera specializes in securing AI-native systems against such threats, offering Lakera Guard as a solution for protecting MCP-based systems without disrupting development. Lakera Guard can detect threats like prompt injections and data leaks with minimal latency by analyzing inputs and outputs in real-time. Implementing security measures in MCP servers is straightforward, requiring just a single line of code, and can be applied to various server components such as tools, prompts, and resources. The example provided demonstrates how to secure a simple MCP server using Lakera Guard, illustrating that a single API call can make the difference between a secure application and a potential failure.
Nov 14, 2025
1,435 words in the original blog post.
Adversarial Machine Learning (AML) is a critical field in artificial intelligence and cybersecurity, focusing on identifying vulnerabilities in AI systems and developing defenses against deceptive tactics that can cause these systems to make significant errors. As AI technologies become more integrated into daily life, the threat of adversarial attacks, which involve crafting inputs to mislead models, poses a serious challenge. This includes potential manipulation of both language models and computer vision systems. The evolution of adversarial tactics highlights the need for continued vigilance and innovative security measures, such as adversarial training, input validation, and real-time anomaly detection, to maintain the integrity and trustworthiness of AI systems. Key figures like Ian Goodfellow emphasize the importance of addressing these challenges to safeguard AI's role in society. Companies like Lakera are actively developing tools like Lakera Guard to protect against advanced adversarial threats, ensuring AI applications remain secure and reliable.
Nov 13, 2025
2,481 words in the original blog post.
Data Loss Prevention (DLP) has become crucial in the era of Generative AI (GenAI), where traditional methods struggle to address new risks associated with AI-driven environments. As organizations increasingly use GenAI tools, the risk of unintentional data leakage grows, requiring modern DLP solutions that go beyond mere compliance to become strategic necessities. These solutions must understand language and context, support workflows involving large language models (LLMs), and provide real-time visibility into data movement. Unlike traditional DLP systems that rely on static data patterns, next-gen DLP tools need to operate at a semantic level, understanding the meaning behind data and preventing leaks through language-based transformations such as summarization, paraphrasing, and translation. The shift to GenAI requires DLP to adapt by offering sensitive data classification, customizable policy enforcement, and real-time monitoring, ultimately redefining data protection to meet the complex demands of today's AI-driven data landscapes.
Nov 13, 2025
1,700 words in the original blog post.
In-context learning (ICL) is a novel approach used in large language models (LLMs) that allows these models to perform new tasks using natural language prompts without the need for explicit retraining. Unlike traditional machine learning, which requires updating model parameters, ICL leverages pre-trained knowledge to generalize from a few input-output examples, a method often referred to as few-shot learning. This technique is enhanced by prompt engineering, which involves crafting effective prompts to guide the model's understanding and output. ICL's efficacy is closely linked to the model's scale, training data quality, and the specific domain it is applied to, showing competitive performance across various applications like sentiment analysis, language translation, and medical diagnostics. Despite its advantages, ICL faces challenges related to model size, data dependency, and domain specificity, and it presents ethical and security concerns. Research in this area is rapidly evolving, focusing on innovations like structured prompting and understanding the relationship between Transformer attention and gradient descent, which further enhance the capabilities of ICL in LLMs.
Nov 13, 2025
3,442 words in the original blog post.
Lakera has launched the Prompt Injection Test (PINT) Benchmark, designed to evaluate prompt injection protection solutions by providing an objective measure against a representative sample of prompt injection and jailbreak attacks. PINT assesses solutions on their ability to detect true positives and minimize false negatives across a dataset of 3,007 English inputs, which include public and proprietary attack techniques, false positives, and challenges with large documents. The benchmark aims to address the complexities in evaluating Generative AI (GenAI) performance, especially in the domain of prompt injection, and invites researchers and solution providers to contribute to and improve its dataset and evaluation methods. The benchmark's results, code, and examples are publicly available under the MIT license, though the dataset is restricted to prevent overfitting. Lakera emphasizes collaboration to enhance the robustness and comprehensiveness of this evaluation tool.
Nov 13, 2025
1,225 words in the original blog post.
Agentic AI systems, which possess both short and long-term memory capabilities, are increasingly susceptible to sophisticated threats like memory poisoning and long-horizon goal hijacks. Memory poisoning involves attackers embedding malicious content into an AI's memory, influencing future actions by recalling these poisoned entries, while long-horizon goal hijacks gradually alter an AI’s objectives to align with an attacker's goals. These threats are persistent and often unnoticed, requiring defenses that treat memory as untrusted input, monitor workflows over time, and employ layered guardrails to mitigate risks. Real-world analogues such as business logic exploits and the Volkswagen emissions scandal illustrate the potential impact of these attacks, highlighting the need for continuous validation of AI actions and objectives. Lakera's research and tools like the Agent Breaker challenge exemplify how simulating these attacks can help organizations build secure AI systems, emphasizing the importance of proactive security measures in protecting against these emerging vulnerabilities.
Nov 13, 2025
1,877 words in the original blog post.
Large Language Models (LLMs) such as GPT-4, Claude, and Gemini are transforming industries with their advanced text and code generation capabilities, but deploying them requires careful consideration of security, privacy, and cost-effectiveness. This comprehensive guide emphasizes the importance of understanding the infrastructure needs of LLMs, which demand significant processing power and data storage, and outlines best practices for their secure deployment. Key strategies include conducting pre-deployment security assessments, employing red teaming to identify vulnerabilities, and using tools like Lakera Red for continuous threat monitoring. The guide also covers the importance of data privacy, recommending techniques like differential privacy and secure multi-party computation to protect sensitive information during model training. Additionally, it discusses the choice between custom and commercial LLMs, highlighting the trade-offs in terms of control and security. Post-deployment, continuous monitoring, patch management, and robust user authentication are vital to maintaining security. The guide concludes by stressing the need for a security-first culture and ongoing vigilance to effectively manage the unique challenges posed by LLMs in real-world applications.
Nov 13, 2025
4,272 words in the original blog post.
Machine learning model monitoring is essential for maintaining the effectiveness, accuracy, and ethical operation of models in real-world applications. It involves observing model performance, detecting issues such as data and model drift, and ensuring models adapt to new data and conditions. Monitoring encompasses various metrics, including stability, performance, and operational factors, to provide a comprehensive understanding of a model's health. The process is not only about technical oversight but also about ensuring compliance with legal and ethical standards, especially as AI systems become more integrated into daily life. Tools like Lakera Guard and Prometheus offer tailored solutions for different monitoring needs, from security in Large Language Models to robust data visualization. Effective monitoring strategies include automation, diverse metric assessment, and regular human review, ensuring that AI systems remain trustworthy and beneficial. Ultimately, proactive monitoring is crucial for the sustainable and responsible deployment of AI technologies across industries.
Nov 13, 2025
3,452 words in the original blog post.
As artificial intelligence (AI) systems become more integral to various industries, aligning them with human values and ethics, known as AI alignment, is crucial to prevent potentially harmful outcomes and maintain trust. AI alignment involves encoding human values into large language models to ensure these systems operate safely and ethically, aligning with societal expectations and human intentions. Techniques such as imitation learning, value learning, and personalized alignment are explored to achieve this alignment, addressing challenges like ambiguity in human values, the attribution problem, and value drift. Despite these challenges, ongoing research aims to enhance AI's ability to align with human goals, while tools like Lakera Guard offer real-time protection against specific AI threats, ensuring AI systems remain secure and reliable.
Nov 13, 2025
1,904 words in the original blog post.
Multimodal large language models (LLMs) have expanded beyond text to process audio, images, and video, enhancing user experiences and enabling new product possibilities, but also introducing significant security challenges. While traditional text-based LLMs had a singular attack vector through user input, multimodal models face a wider range of threats due to their ability to interpret nuanced audio inputs, making them susceptible to various acoustic attacks. These attacks include methods like clean audio jailbreaks, transcriber bypass via reverberation, dual-audio obfuscation, and transcriber muting, which exploit the limitations of transcription-based defenses. To combat these vulnerabilities, Lakera Guard provides an advanced security solution by analyzing raw audio streams for adversarial patterns and malicious intents, operating independently of transcription quality, thereby offering real-time protection against these evolving threats. As multimodal systems grow more prevalent, effective security measures like those provided by Lakera Guard become crucial in mitigating risks associated with their expanded attack surfaces.
Nov 13, 2025
1,223 words in the original blog post.
The rapid evolution and adoption of generative AI have prompted significant regulatory developments in the EU, US, and UK, focusing on AI safety and security. The EU is advancing its AI Act, with Article 28b emphasizing responsible risk management for AI systems, while the US has issued an Executive Order to promote safe and trustworthy AI through collaboration between government and private sectors. The UK's AI Safety Summit further underscores the need for international cooperation to mitigate AI risks, particularly concerning frontier AI technologies. These regulatory efforts highlight the importance for businesses to prepare for compliance, advocating for increased testing, adoption of open standards, and a secure-by-design approach that integrates ethical considerations. Companies like Lakera are actively engaging in these regulatory discussions, contributing insights and solutions to help organizations navigate this complex landscape and ensure responsible AI innovation.
Nov 13, 2025
1,589 words in the original blog post.
Remote Code Execution (RCE) is a significant cybersecurity threat that enables attackers to execute arbitrary code on a target system without prior access, potentially leading to full system compromise, data theft, and the spread of malware. The rise of interconnected devices has increased the risk of RCE attacks, which exploit software vulnerabilities like improper input validation and buffer overflows. High-profile vulnerabilities, such as Log4Shell, demonstrate the widespread impact and ease of exploitation that can affect millions of systems globally. Organizations are advised to adopt a multi-faceted approach to mitigate RCE risks, including regular software updates, secure coding practices, vulnerability scanning, and penetration testing. The integration of AI and Large Language Models (LLMs) in cybersecurity introduces both opportunities and vulnerabilities, with prompt injection attacks posing a particular threat to AI systems. Rigorous security measures, such as input validation and layered security architectures, are essential to protect against these threats. As AI technologies continue to evolve, the community must balance algorithmic accuracy with ethical considerations to ensure the safe and responsible deployment of AI systems.
Nov 13, 2025
4,013 words in the original blog post.
Shadow AI, a rising trend in organizations, refers to the unsanctioned use of generative AI tools without oversight from IT departments, often driven by the accessibility of technologies like ChatGPT. This phenomenon presents numerous challenges, including data privacy concerns, non-compliance with regulations, and security vulnerabilities due to the dynamic nature of AI models. Companies are encouraged to adopt proactive strategies to manage shadow AI, balancing innovation with risk management through active monitoring, technical controls, and clear governance policies. Lakera offers solutions like the Lakera Chrome Extension and Lakera Guard to address these risks, focusing on securing AI applications, particularly Large Language Models (LLMs), by implementing threat monitoring, access control, and vulnerability management. As the use of AI continues to grow, organizations must shift from prevention to proactive management, fostering a culture of responsible and informed AI usage to maximize benefits while mitigating risks.
Nov 13, 2025
2,880 words in the original blog post.
"Securing AI Agents in Production: A Practical Guide" serves as a comprehensive resource for understanding and addressing the security challenges associated with deploying AI technologies, specifically focusing on large language models (LLMs) and autonomous generative AI systems. It covers a range of topics, including a detailed analysis of vulnerabilities in LLM applications, insights from a vast database of LLM attack data, and practical security measures such as data sanitization and PII detection. The guide introduces Gandalf, an educational game for AI security, and outlines Lakera Guard, a solution designed to counter common AI threats. It emphasizes the importance of designing secure AI agents by incorporating guardrails, tool restrictions, and robust prompt architectures, while highlighting the limitations of static filters and the necessity of real-time security measures. Through real-world examples and case studies like Dropbox's AI agent security, the guide offers actionable strategies for building, monitoring, and defending AI applications, advocating for proactive measures to prevent security incidents as AI agents transition from prototypes to production.
Nov 13, 2025
386 words in the original blog post.
In 2023, Large Language Models (LLMs) transformed the AI landscape with their advanced comprehension capabilities, becoming essential tools for solving complex issues beyond mere content generation. As their influence grows across various industries, rigorous evaluation becomes crucial to ensure their reliability, accuracy, safety, and fairness. Evaluating LLMs involves assessing contextual comprehension and bias neutrality, among other factors, using diverse methods and frameworks. These evaluations highlight strengths and pinpoint areas for improvement, guiding developers toward enhancing models. Challenges in current evaluation techniques include granularity of metrics, overfitting to benchmarks, and the need for diverse testing data, but best practices such as employing diverse datasets, multi-faceted evaluation, and real-world testing are recommended to address these. As AI and NLP continue to evolve, future evaluation methodologies will increasingly focus on context, emotional resonance, and ethical considerations, emphasizing the importance of adaptable and ethically grounded assessments to guide future advancements in the field.
Nov 13, 2025
4,592 words in the original blog post.
Artificial intelligence (AI) is revolutionizing industries, but its increasing complexity poses challenges in ensuring responsible and ethical use. AI observability emerges as a key solution, offering a deeper understanding of AI models beyond traditional monitoring by examining their decision-making processes, data usage, and performance over time. This comprehensive approach is crucial for building trust and mitigating risks, particularly in generative AI systems that present both innovative opportunities and potential pitfalls. Observability practices are integrated throughout the AI lifecycle, from problem definition to deployment, ensuring models remain effective, reliable, and ethical. Challenges such as data drift, model complexity, and explainability require careful management, while observability tools and strategies like LLM-specific solutions and risk management frameworks are evolving to address these issues. By prioritizing observability, organizations can enhance AI reliability and transparency, gaining a competitive edge and ensuring AI's positive societal impact.
Nov 13, 2025
4,573 words in the original blog post.
The text discusses the historical challenges and lessons learned from attempts to hard-code intelligence into machines, highlighting the shift from manual rule-based systems to machine learning and deep learning models, such as AlexNet, which revolutionized fields like image recognition and natural language processing by using data-driven approaches. It draws parallels to the current state of AI security, critiquing the reliance on outdated methods like pattern matching and static guardrails that fail to address the dynamic and complex nature of AI threats. The author argues for an AI-native approach to security, advocating for adaptive models that can recognize and respond to novel attacks, emphasizing the need for rapid adaptation to avoid repeating past mistakes and urging the application of machine learning principles to build robust AI defenses.
Nov 13, 2025
901 words in the original blog post.
Large language models (LLMs) like GPT and Llama have transformed technological interactions, but their limitations in accuracy and context retention have led to the development of Retrieval Augmented Generation (RAG). RAG enhances LLM performance by integrating external retrieval systems that provide contextually relevant and updated information, akin to a student consulting a textbook during a test. This method involves two key components: a retriever that locates pertinent data using techniques like dense retrieval and semantic search, and a generator that crafts coherent responses based on this data. While RAG offers advantages such as reduced training costs, enhanced scalability, and access to diverse knowledge sources, it also faces challenges including potential inaccuracies, scalability issues, and biases in data retrieval. Despite these challenges, RAG's application across industries like healthcare, finance, and customer support demonstrates its value in providing precise, real-time information, thereby enhancing decision-making and interaction quality.
Nov 13, 2025
2,595 words in the original blog post.