Googlebot: Google's search crawler
Google's main crawler for Google Search.
- Free, no account.
- A few seconds.
- Nothing is stored.
Googlebot at a glance
Everything Google documents about it, with links to the source.
- Operator
- Type
- Search engines
- Used for
- Google Search, including AI Overviews and AI Mode
- Obeys robots.txt
- Yes. Google documents that Googlebot follows robots.txt, so a rule for its token is honoured.
- robots.txt token
Googlebot- User-agent string
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- Reverse DNS
- Resolves to *.googlebot.com, *.google.com
How to block Googlebot
Crawls for a search engine's index. Blocking it removes your pages from that search engine, and from the AI answers built on it (AI Overviews on Googlebot, Copilot on Bingbot).
Block the whole site
Add this group to robots.txt at the root of your site.
User-agent: Googlebot Disallow: /
Block only some paths
List the paths; everything else stays open to Googlebot.
User-agent: Googlebot Disallow: /private/ Disallow: /drafts/
Careful: blocking Googlebot removes the site from Google Search, AI Overviews and AI Mode. To keep content out of Gemini training only, block Google-Extended.
How to verify Googlebot
Anyone can send a crawler's user-agent string. Where the request comes from is what proves it.
- 1
Find requests whose user agent contains "Googlebot" in your server or CDN logs.
- 2
Check each source IP against Google's published list (developers.google.com/static/search/apis/ipranges/googlebot.json). A request from outside it is not Googlebot, whatever it says.
- 3
Run a reverse DNS lookup on the IP. A genuine request resolves to a host under googlebot.com or google.com, and a forward lookup of that host returns the same IP.
- 4
Block what fails the check at the firewall. A robots.txt rule only reaches crawlers that choose to read it.
Other crawlers from Google
Each has its own token, and blocking one leaves the others untouched.
Frequently asked questions
Google's main crawler for Google Search. What it fetches is also what AI Overviews and AI Mode draw on, since both are part of Search. It crawls mostly as a smartphone, and the string above is one of several it sends.
Yes. Google documents that Googlebot follows robots.txt, so a rule for its token is honoured.
Add a group for its token to robots.txt at the root of your site: "User-agent: Googlebot" followed by "Disallow: /". To close only part of the site, list those paths instead of "/". To enforce the block rather than request it, deny the IP addresses Google publishes at your firewall.
Yes. AI Overviews and AI Mode are part of Google Search and use what Googlebot crawls, so blocking Googlebot removes your pages from both, along with Search itself. To opt out of Gemini training only, block Google-Extended.
Checked against Google's documentation on 29 September 2026.
Next: see whether AI recommends you
Letting the right crawlers in is the first step. AskWatch shows what ChatGPT, Perplexity, Gemini and Google AI answer when buyers ask about your category, and who they name.
- Free, no credit card.
- Report in minutes, link sent to your email.