Google Documents Its Three Types Of Web Crawlers

Apr 24, 2023 - 6:31 am 0 by

Google Bot Lineup

Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers.

I believe this was done after we, including me, were obsessed a bit over the new GoogleOther crawler. Then Gary Illyes from Google added, "Please don't overthink it, it's really that boring." But I do what I do and I overthink'ed it. So Gary did what he does and had a help document to explain this in more detail.

The help document says, "Google's crawlers fall into three categories."

(1) Googlebot: The main crawler for Google's search products, it alays respects robots.txt rules. Its reverse DNS mask is "crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com" and the list of IP ranges are in this googlebot.json file.

(2) Special-case crawlers: Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules. Its reverse DNS mask is "rate-limited-proxy-***-***-***-***.google.com" and the list of IP ranges are in this special-crawlers.json file.

(3) User-triggered fetchers: Tools and product functions where the end user triggers a fetch. For example, Google Site Verifier acts on the request of a user. Because the fetch was requested by a user, these fetchers ignore robots.txt rules. Its reverse DNS mask is "***-***-***-***.gae.googleusercontent.com" and the list of IP ranges are in this user-triggered-fetchers.json file.

Here is a screenshot of the new section in this help document:

click for full size

Also, with this, Google restructured the order of the Googlebots on the Google crawlers page:

Old New Googlebot Structured Order

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Video Recaps

Search News Buzz Video Recap: Google Helpful Content System, Reddit’s Google Deal, Product Variant Schema, Google Ads, Bing Search & Search Volume Predictions

Feb 23, 2024 - 8:01 am
Google Ads

Google Ad Strength For PMAX To Weigh Assets Quantity & Variety More As AI Expands

Feb 23, 2024 - 7:51 am
Google Search Engine Optimization

Google: You Decide What Old Content Is Helpful Or Not

Feb 23, 2024 - 7:41 am
Google

No Joke: Google Is Paying Reddit For More Content, More Often

Feb 23, 2024 - 7:31 am
Google Ads

Google Ads API Version 16 Now Available

Feb 23, 2024 - 7:21 am
Bing Search

Bing Search Yellow Opinion Label On News Stories

Feb 23, 2024 - 7:11 am
Previous Story: Bing Chat Better At News / Sports & Google Bard Can Code, Plus More AI Updates