Say it once: introducing Bot Preference Sync

Cloudflare is introducing Bot Preference Sync, a feature available across all plans that automatically updates a website’s robots.txt file to match AI bot policies configured in its dashboard for Search, Agent, and Training traffic. The tool is intended to reduce conflicts between stated crawler preferences and edge-enforced bot controls, while allowing site owners to choose policies that fit their business models, such as maximizing AI-driven discovery or preventing model training on ad-supported content. Training can be disallowed without necessarily affecting search indexing for verified mixed-use crawlers that meet Cloudflare’s transparency requirements, including honoring no-training directives, providing opt-outs from AI summaries, and offering page-level usage visibility. Bot Preference Sync prepends Cloudflare-managed directives to existing robots.txt content and updates bot lists using Cloudflare’s verified bot data, though customers with specialized exceptions or custom rules can disable it and manage their files independently. New publishing customers can select an ad-monetization option that defaults to disallowing training, while other new customers receive no default blocks, reflecting Cloudflare’s broader emphasis on transparency, consent, discoverability, and site-owner control over AI crawler access.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.