GPTBot: OpenAI's AI training crawler
OpenAI's crawler for content that may be used to train its generative AI foundation models.
- Free, no account.
- A few seconds.
- Nothing is stored.
GPTBot at a glance
Everything OpenAI documents about it, with links to the source.
- Operator
- OpenAI
- Type
- AI training
- Used for
- Training OpenAI's models
- Obeys robots.txt
- Yes. OpenAI documents that GPTBot follows robots.txt, so a rule for its token is honoured.
- robots.txt token
GPTBot- User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
- IP addresses
- openai.com/gptbot.json
- Documentation
- developers.openai.com/api/docs/bots
How to block GPTBot
Collects text that may be used to train AI models. Blocking it keeps your pages out of future training data. AI answers can still cite you through the search crawlers.
Block the whole site
Add this group to robots.txt at the root of your site.
User-agent: GPTBot Disallow: /
Block only some paths
List the paths; everything else stays open to GPTBot.
User-agent: GPTBot Disallow: /private/ Disallow: /drafts/
How to verify GPTBot
Anyone can send a crawler's user-agent string. Where the request comes from is what proves it.
- 1
Find requests whose user agent contains "GPTBot" in your server or CDN logs.
- 2
Check each source IP against OpenAI's published list (openai.com/gptbot.json). A request from outside it is not GPTBot, whatever it says.
- 3
Block what fails the check at the firewall. A robots.txt rule only reaches crawlers that choose to read it.
Other crawlers from OpenAI
Each has its own token, and blocking one leaves the others untouched.
Frequently asked questions
OpenAI's crawler for content that may be used to train its generative AI foundation models. Blocking it keeps a site out of future training data; it does not affect whether ChatGPT search shows the site, which is OAI-SearchBot's job.
Yes. OpenAI documents that GPTBot follows robots.txt, so a rule for its token is honoured.
Add a group for its token to robots.txt at the root of your site: "User-agent: GPTBot" followed by "Disallow: /". To close only part of the site, list those paths instead of "/". To enforce the block rather than request it, deny the IP addresses OpenAI publishes at your firewall.
No. GPTBot is a training crawler: blocking it keeps your pages out of data that may train OpenAI's models. AI answers that cite pages come from search crawlers, which have their own tokens, so blocking GPTBot alone does not remove you from them.
Checked against OpenAI's documentation on 29 September 2026.
Next: see whether AI recommends you
Letting the right crawlers in is the first step. AskWatch shows what ChatGPT, Perplexity, Gemini and Google AI answer when buyers ask about your category, and who they name.
- Free, no credit card.
- Report in minutes, link sent to your email.