Claude Code ROBOTS_DISALLOWED: Why WebFetch Cannot Read Some Links

ROBOTS_DISALLOWED in Claude Code’s WebFetch indicates that a request was permitted locally but refused by a website’s robots.txt policy for the Claude-User agent, rather than signaling a broken installation, private page, or universal AI restriction. A reported October 2026 issue involved x.com, api.fxtwitter.com, and TikTok, whose robots rules reportedly blocked the relevant agent, although the available evidence did not establish whether the change originated with site operators, Anthropic, or a specific Claude Code version. The guidance recommends distinguishing robots refusals from tool availability, local permissions, network preflight checks, redirects, timeouts, and lossy extraction by reviewing raw tool results, testing a known public page, checking the target host’s robots.txt, and recording environment details. When access is denied, users should seek and cite primary sources such as official documentation, release notes, repositories, or publisher announcements; clearly label search snippets and user-provided text; and avoid attempts to bypass site policies. Website operators who want to permit user-initiated Claude access can configure robots.txt for Claude-User, while alternative retrieval tools or MCP servers may provide separate, policy-dependent ways to locate authoritative sources without repairing WebFetch itself.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.