Google Documents Its Three Types Of Web Crawlers

Apr 24, 2023 - 7:31 am 0 — by

Google Bot Lineup

Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers.

I believe this was done after we, including me, were obsessed a bit over the new GoogleOther crawler. Then Gary Illyes from Google added, "Please don't overthink it, it's really that boring." But I do what I do and I overthink'ed it. So Gary did what he does and had a help document to explain this in more detail.

The help document says, "Google's crawlers fall into three categories."

(1) Googlebot: The main crawler for Google's search products, it alays respects robots.txt rules. Its reverse DNS mask is "crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com" and the list of IP ranges are in this googlebot.json file.

(2) Special-case crawlers: Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules. Its reverse DNS mask is "rate-limited-proxy-***-***-***-***.google.com" and the list of IP ranges are in this special-crawlers.json file.

(3) User-triggered fetchers: Tools and product functions where the end user triggers a fetch. For example, Google Site Verifier acts on the request of a user. Because the fetch was requested by a user, these fetchers ignore robots.txt rules. Its reverse DNS mask is "***-***-***-***.gae.googleusercontent.com" and the list of IP ranges are in this user-triggered-fetchers.json file.

Here is a screenshot of the new section in this help document:

click for full size

Also, with this, Google restructured the order of the Googlebots on the Google crawlers page:

Old New Googlebot Structured Order

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google

Google Search Officially Rolls Out Collapsible Sidebar

Nov 19, 2026 - 2:38 pm
Bing Ads

Microsoft Confirms Ads Are In AI Overviews On Bing Search

Nov 19, 2026 - 10:24 am
Search Forum Recap

Daily Search Forum Recap: November 19, 2026

Nov 19, 2026 - 10:00 am
Google

Google Search What People Are Saying AI-Generated Emojis

Nov 19, 2026 - 7:51 am
Google

Google Hotel Search Listings With Continue On Google To Book

Nov 19, 2026 - 7:41 am
Google

Google Search Tests "Show Original Results" Button

Nov 19, 2026 - 7:31 am
Previous Story: Bing Chat Better At News / Sports & Google Bard Can Code, Plus More AI Updates