Datadog has integrated large language models (LLMs) into its security workflows to enhance the detection of malicious code changes in pull requests (PRs) at scale, addressing challenges such as growing attack surfaces and reviewer fatigue. This LLM-powered system, named BewAIre, analyzes the intent behind code modifications and provides real-time security signals for Datadog's repositories, achieving over 99.3% accuracy with a minimal false positive rate. By employing prompt engineering, dataset tuning, and recursive chunking of large diffs, Datadog has developed a system that effectively identifies sophisticated attacks, such as the tj-actions breach, by understanding the intent behind code changes rather than just syntax. This initiative represents a shift from traditional static analysis tools towards smarter scanning methods that integrate seamlessly into developer workflows, highlighting the practical application of LLMs as a defense layer against novel attack vectors. Additionally, Datadog plans to expand its security measures to include detection of insecure code markers like personally identifiable information, indicating a broader vision for integrating AI-driven insights into its security posture.