AI lectures with Dr. Brumley Part 2 | The anatomy of a modern AI system

In a recent AI lecture series based on a talk at Carnegie Mellon University, the complexities and potential vulnerabilities of modern AI systems were explored, emphasizing the importance for Chief Information Security Officers (CISOs) to understand these aspects for effective security governance. AI systems, often treated as black boxes, should instead be dissected into their core training stages—pretraining, Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF)—each presenting unique security risks such as corpus and demonstration poisoning. The lecture highlighted how the context window at inference time poses significant security concerns since it includes unstructured text from various sources, which can be misinterpreted as instructions, thus lacking a structural trust boundary. Furthermore, the integration of tools and agents in LLM deployments expands the threat model, allowing models to perform actions that could result in adversarial outcomes. Governance requires careful consideration of training data provenance, model supply chain integrity, context window composition, tool permissions, and agent autonomy settings. These considerations are crucial for organizations seeking to mitigate risks before regulatory demands intensify, with future discussions in the series set to address securing the AI attack surface.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.