Turning OWASP AILLM Risks into Practical QA Checks [Testμ 2026]

At Testμ Conf 2026, Thoughtworks quality analyst Thejes Sree Satheesh Kumar argued that LLM-enabled products should be tested through the observable systems around them—such as retrieval records, prompts, tool actions, rendered output, and costs—rather than by trying to validate the model’s wording or universal truthfulness. Using a fictional support chatbot that refunds $4,900 after interpreting a malicious product review as an instruction, he connected OWASP LLM security risks including prompt injection, excessive agency, data disclosure, misinformation, and unsafe output handling to measurable behavioral checks. His proposed process is to select one risk, define a harmful scenario, create an adversarial probe, identify a concrete signal such as a refund amount or retrieved private document, repeat the test several times, and apply risk-based thresholds, with financial and personal-data controls expected to succeed every time. He also recommended monitoring pass-rate drift to identify unannounced model changes and running checks in fast pull-request suites, nightly real-model runs, and human-approved pre-release tests. The session emphasized logging, action restrictions, approval limits, output escaping, and retrieval controls, while acknowledging that the demonstration did not provide source code, tool details, configurations, log schemas, or reproducible guardrail implementations.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.