Google Documents Its Three Types Of Web Crawlers

Apr 24, 2023 - 7:31 am 0 by

Google Bot Lineup

Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers.

I believe this was done after we, including me, were obsessed a bit over the new GoogleOther crawler. Then Gary Illyes from Google added, "Please don't overthink it, it's really that boring." But I do what I do and I overthink'ed it. So Gary did what he does and had a help document to explain this in more detail.

The help document says, "Google's crawlers fall into three categories."

(1) Googlebot: The main crawler for Google's search products, it alays respects robots.txt rules. Its reverse DNS mask is "crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com" and the list of IP ranges are in this googlebot.json file.

(2) Special-case crawlers: Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules. Its reverse DNS mask is "rate-limited-proxy-***-***-***-***.google.com" and the list of IP ranges are in this special-crawlers.json file.

(3) User-triggered fetchers: Tools and product functions where the end user triggers a fetch. For example, Google Site Verifier acts on the request of a user. Because the fetch was requested by a user, these fetchers ignore robots.txt rules. Its reverse DNS mask is "***-***-***-***.gae.googleusercontent.com" and the list of IP ranges are in this user-triggered-fetchers.json file.

Here is a screenshot of the new section in this help document:

click for full size

Also, with this, Google restructured the order of the Googlebots on the Google crawlers page:

Old New Googlebot Structured Order

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google: Some SEOs Over Focus On URL Structure

Nov 13, 2024 - 7:51 am
Google Ads

Google Customer Match Policy Warning: Don't Harm Users Or Else...

Nov 13, 2024 - 7:41 am
Google AdSense

Google AdSense First-Party Cookie Controls Are Changing

Nov 13, 2024 - 7:31 am
Google Search Engine Optimization

New: Google Search Supports C2PA Metadata For About This Image Feature

Nov 13, 2024 - 7:21 am
Google Search Engine Optimization

Google Merchant Center Adds Sale Events Promotion Type

Nov 13, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: November 12, 2024

Nov 12, 2024 - 10:00 am
Previous Story: Bing Chat Better At News / Sports & Google Bard Can Code, Plus More AI Updates