Robotcop: enforcing your robots.txt policies and stopping bots before they reach your website

Cloudflare's AI Audit dashboard now allows users to quickly see which AI services are honoring their robots.txt policies, identify non-compliant services, and programmatically enforce these policies at the network level.` This new feature provides visibility into how AI companies and services access content, helps customers define and enforce robots.txt policies, and reduces automated traffic to servers. The feature is live today for all Cloudflare customers, who can log in to their dashboard and navigate to their domain to begin auditing bot traffic from AI services and enforcing robots.txt directives.