Teach Your robots.txt a New Trick (for AI)
Blog post from Fastly
Recently, both Google and Apple introduced new user agents that allow website owners to control whether their content is used to train AI models by updating their robots.txt files. Google-Extended and Applebot-Extended are special instructions within the robots.txt file that prevent the use of website content for AI training while not affecting search rankings. This gives website owners the power to protect intellectual property by opting out of contributing to AI training data. Implementing these changes involves simple modifications to the robots.txt file, allowing for greater control over content usage. Additionally, tools like Fastly Bot Management offer insights into various bots accessing web content, providing further control and visibility.
No tracked trend matches for this post yet.
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.