meta-externalfetcher: Meta's on-demand fetcher
Meta's fetcher for individual links at a user's request, including helping AI navigate websites to complete tasks for users.
- Free, no account.
- A few seconds.
- Nothing is stored.
meta-externalfetcher at a glance
Everything Meta documents about it, with links to the source.
- Operator
- Meta
- Type
- Fetches on a user's request
- Used for
- Links a user asks a Meta AI product to fetch
- Obeys robots.txt
- Not always. Meta says that because meta-externalfetcher fetches pages a user asked for, robots.txt rules may not apply. A rule still states your preference; to enforce it, block the published IP addresses at your firewall.
- robots.txt token
meta-externalfetcher- User-agent string
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
- IP addresses
- Not published.
How to block meta-externalfetcher
Opens a page because a person asked an assistant to read it. Blocking it stops those visits for the operators that honour robots.txt. Several say they may not, so a firewall rule is the real block.
Block the whole site
Add this group to robots.txt at the root of your site.
User-agent: meta-externalfetcher Disallow: /
Block only some paths
List the paths; everything else stays open to meta-externalfetcher.
User-agent: meta-externalfetcher Disallow: /private/ Disallow: /drafts/
robots.txt is a request meta-externalfetcher may not follow. To enforce it, block its requests at your server or firewall.
How to verify meta-externalfetcher
Anyone can send a crawler's user-agent string. Where the request comes from is what proves it.
- 1
Find requests whose user agent contains "meta-externalfetcher" in your server or CDN logs.
- 2
Meta publishes no IP list for meta-externalfetcher, so it cannot be verified from the request alone. Treat unusual volume as a scraper using its name, and rate-limit or block it at the firewall.
- 3
Block what fails the check at the firewall. A robots.txt rule only reaches crawlers that choose to read it.
Other crawlers from Meta
Each has its own token, and blocking one leaves the others untouched.
Frequently asked questions
Meta's fetcher for individual links at a user's request, including helping AI navigate websites to complete tasks for users. Meta says that accordingly it may bypass robots.txt rules.
Not always. Meta says that because meta-externalfetcher fetches pages a user asked for, robots.txt rules may not apply. A rule still states your preference; to enforce it, block the published IP addresses at your firewall.
Add a group for its token to robots.txt at the root of your site: "User-agent: meta-externalfetcher" followed by "Disallow: /". To close only part of the site, list those paths instead of "/".
Because Meta treats these fetches as a user's own request and says robots.txt may not apply. If you need to stop them, block the crawler's IP addresses at the server or firewall.
Checked against Meta's documentation on 29 September 2026.
Next: see whether AI recommends you
Letting the right crawlers in is the first step. AskWatch shows what ChatGPT, Perplexity, Gemini and Google AI answer when buyers ask about your category, and who they name.
- Free, no credit card.
- Report in minutes, link sent to your email.