Claude-User: Anthropic's on-demand fetcher
Claude's fetcher for pages a user's question leads it to.
- Free, no account.
- A few seconds.
- Nothing is stored.
Claude-User at a glance
Everything Anthropic documents about it, with links to the source.
- Operator
- Anthropic
- Type
- Fetches on a user's request
- Used for
- Pages a Claude user asks it to open
- Obeys robots.txt
- Yes. Anthropic documents that Claude-User follows robots.txt, so a rule for its token is honoured.
- robots.txt token
Claude-User- User-agent string
- Anthropic does not publish the full string. Match requests on the token Claude-User, and verify them against the IP list below.
- IP addresses
- claude.com/crawling/bots.json
- Documentation
- support.claude.com/en/articles/8896518
How to block Claude-User
Opens a page because a person asked an assistant to read it. Blocking it stops those visits for the operators that honour robots.txt. Several say they may not, so a firewall rule is the real block.
Block the whole site
Add this group to robots.txt at the root of your site.
User-agent: Claude-User Disallow: /
Block only some paths
List the paths; everything else stays open to Claude-User.
User-agent: Claude-User Disallow: /private/ Disallow: /drafts/
How to verify Claude-User
Anyone can send a crawler's user-agent string. Where the request comes from is what proves it.
- 1
Find requests whose user agent contains "Claude-User" in your server or CDN logs.
- 2
Check each source IP against Anthropic's published list (claude.com/crawling/bots.json). A request from outside it is not Claude-User, whatever it says.
- 3
Block what fails the check at the firewall. A robots.txt rule only reaches crawlers that choose to read it.
Other crawlers from Anthropic
Each has its own token, and blocking one leaves the others untouched.
Frequently asked questions
Claude's fetcher for pages a user's question leads it to. Anthropic says it respects robots.txt, and that blocking it stops Claude from retrieving a site's content for a user's query, which may reduce the site's visibility in user-directed web search.
Yes. Anthropic documents that Claude-User follows robots.txt, so a rule for its token is honoured.
Add a group for its token to robots.txt at the root of your site: "User-agent: Claude-User" followed by "Disallow: /". To close only part of the site, list those paths instead of "/". To enforce the block rather than request it, deny the IP addresses Anthropic publishes at your firewall.
It should not: Anthropic documents that Claude-User follows robots.txt. Check the rule with our tester, and remember that anyone can send its user-agent string; compare the request's IP with Anthropic's published list.
Checked against Anthropic's documentation on 29 September 2026.
Next: see whether AI recommends you
Letting the right crawlers in is the first step. AskWatch shows what ChatGPT, Perplexity, Gemini and Google AI answer when buyers ask about your category, and who they name.
- Free, no credit card.
- Report in minutes, link sent to your email.