Company
Date Published
Author
João Tomé
Word count
2269
Language
English
Hacker News points
None

Summary

In 2025, the landscape of internet usage and content consumption is being significantly reshaped by generative AI, particularly through the increased activity of AI training crawlers that consume large volumes of web data while contributing less referral traffic back to content creators. This shift is evidenced by declining referral traffic from traditional search engines like Google, which now features AI-generated summaries at the top of search results, contributing to reduced visits to news websites and impacting advertising revenues and subscription conversions. Data from Cloudflare and other studies indicate that AI-related crawling activity surged by 32% year-over-year in April 2025, though growth slowed to 4% by July. Training-related crawling has become dominant, accounting for nearly 80% of AI bot activity, as AI models like OpenAI's GPTBot and Anthropic's ClaudeBot increase their presence, while others like ByteDance's Bytespider see declines. The imbalance between crawling and referral traffic highlights the challenge publishers face in this new ecosystem, as they provide data for AI training but receive minimal traffic in return, posing a potential threat to content creation sustainability unless new models of compensation or collaboration are developed.