Google Documents Its Three Types Of Web Crawlers

Apr 24, 2023 - 7:31 am 0 by

Google Bot Lineup

Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers.

I believe this was done after we, including me, were obsessed a bit over the new GoogleOther crawler. Then Gary Illyes from Google added, "Please don't overthink it, it's really that boring." But I do what I do and I overthink'ed it. So Gary did what he does and had a help document to explain this in more detail.

The help document says, "Google's crawlers fall into three categories."

(1) Googlebot: The main crawler for Google's search products, it alays respects robots.txt rules. Its reverse DNS mask is "crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com" and the list of IP ranges are in this googlebot.json file.

(2) Special-case crawlers: Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules. Its reverse DNS mask is "rate-limited-proxy-***-***-***-***.google.com" and the list of IP ranges are in this special-crawlers.json file.

(3) User-triggered fetchers: Tools and product functions where the end user triggers a fetch. For example, Google Site Verifier acts on the request of a user. Because the fetch was requested by a user, these fetchers ignore robots.txt rules. Its reverse DNS mask is "***-***-***-***.gae.googleusercontent.com" and the list of IP ranges are in this user-triggered-fetchers.json file.

Here is a screenshot of the new section in this help document:

click for full size

Also, with this, Google restructured the order of the Googlebots on the Google crawlers page:

Old New Googlebot Structured Order

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Ads

Google Ads API Outages and Slowness; Microsoft Ads Warns About Google Imports

Jun 29, 2025 - 8:12 am
Google Updates

Heated Google Search Ranking Volatility On Saturday June 28th

Jun 29, 2025 - 7:59 am
Search Forum Recap

Daily Search Forum Recap: June 27, 2025

Jun 27, 2025 - 10:00 am
Google

Google AI Overviews Tests Showing Site Names

Jun 27, 2025 - 8:10 am
Search Video Recaps

Search News Buzz Video Recap: Where The Google Core Updates, Web Is Thriving & AI Index Freshness

Jun 27, 2025 - 8:01 am
Google

Head Of Google Search, Liz Reid, On Ads & SEO With AI Search

Jun 27, 2025 - 7:51 am
Previous Story: Bing Chat Better At News / Sports & Google Bard Can Code, Plus More AI Updates