Have it both ways: stay discoverable in search while disallowing AI training

Cloudflare has introduced a Disallow AI Training setting that allows website owners to remain visible in search indexes while signaling that their content should not be used for AI training, addressing the problem posed by mixed-use crawlers that perform both functions. The setting publishes a robots.txt preference, permits accountable mixed-use crawlers such as Applebot, Googlebot, and Bingbot to continue indexing sites for search, and blocks other training crawlers without affecting search visibility; selecting the broader Block option now stops mixed-use crawlers entirely, including their search activity. Cloudflare also replaces older AI bot controls with separate Search, Training, and Agent controls, migrates most existing settings automatically, and offers more restrictive recommended presets for advertising-supported sites. Apple, Google, and Microsoft have been designated Accountable based on existing or committed controls for AI-training opt-outs, AI-summary exclusions, transparency, and assurances that training choices will not harm search rankings, although Microsoft’s robots.txt support is targeted for early 2027. Looking ahead, Cloudflare plans to develop controls over the amount of website content used in AI summaries, noting that summaries may reduce traffic but can also send higher-intent visitors, making the appropriate choice dependent on each site’s business model.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.