A deeper look at AI crawlers: breaking down traffic by purpose and industry

The traditional model of search platforms driving traffic to websites, which in turn generated ad revenue for publishers, has been disrupted by the rise of AI platforms that provide direct answers without requiring users to click through to source sites. This shift has prompted Cloudflare Radar to introduce new analytics on AI bot and crawler traffic, including crawl/refer ratios and insights into AI bot activities categorized by purpose, such as training, search, and user actions. Notably, the majority of crawling traffic is aimed at gathering data to train AI models, often ignoring standard web directives like robots.txt files. Cloudflare's tools now allow users to explore bot traffic trends by industry, vertical, and purpose, revealing variations in crawling patterns and crawl-to-refer ratios across different sectors. As AI crawlers become more prevalent and their purposes diversify, Cloudflare is working on providing more granular data to help content owners manage this traffic, while also developing standards for how automated systems should interact with web content.