Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

Perplexity, an AI-powered answer engine, has been observed engaging in stealth crawling behavior that circumvents website preferences, despite being initially blocked through their declared user agents. This behavior includes modifying user agents, changing source ASNs, and ignoring robots.txt directives, prompting Cloudflare to de-list them as a verified bot and implement heuristics to block such activity. Tests confirmed that Perplexity accesses content on restricted domains by using undeclared crawlers with rotating IPs and ASNs, contrary to web crawling norms. In contrast, OpenAI is highlighted as a positive example, adhering to best practices by respecting robots.txt and not evading blocks. Cloudflare has developed measures to block Perplexity's stealth crawling and continues to collaborate with experts to establish clear standards for bot behavior, allowing customers to protect their content against unauthorized AI access.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.