Google Documents Its Three Types Of Web Crawlers

Apr 24, 2023 - 7:31 am 0 by

Google Bot Lineup

Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers.

I believe this was done after we, including me, were obsessed a bit over the new GoogleOther crawler. Then Gary Illyes from Google added, "Please don't overthink it, it's really that boring." But I do what I do and I overthink'ed it. So Gary did what he does and had a help document to explain this in more detail.

The help document says, "Google's crawlers fall into three categories."

(1) Googlebot: The main crawler for Google's search products, it alays respects robots.txt rules. Its reverse DNS mask is "crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com" and the list of IP ranges are in this googlebot.json file.

(2) Special-case crawlers: Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules. Its reverse DNS mask is "rate-limited-proxy-***-***-***-***.google.com" and the list of IP ranges are in this special-crawlers.json file.

(3) User-triggered fetchers: Tools and product functions where the end user triggers a fetch. For example, Google Site Verifier acts on the request of a user. Because the fetch was requested by a user, these fetchers ignore robots.txt rules. Its reverse DNS mask is "***-***-***-***.gae.googleusercontent.com" and the list of IP ranges are in this user-triggered-fetchers.json file.

Here is a screenshot of the new section in this help document:

click for full size

Also, with this, Google restructured the order of the Googlebots on the Google crawlers page:

Old New Googlebot Structured Order

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: December 6, 2024

Dec 6, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google November Core Update Done, Chrome Site Engagement Metrics, Canonicals, 21 Years & More

Dec 6, 2024 - 8:11 am
Google Updates

Google November 2024 Core Update Finally Finished Rolling Out

Dec 6, 2024 - 8:01 am
Google Search Engine Optimization

Google Does Try To Handle Broken Canonicals

Dec 6, 2024 - 7:51 am
Google Search Engine Optimization

Google Search: How Clustering Works With Localization

Dec 6, 2024 - 7:41 am
Google Search Engine Optimization

Google Marauding Black Holes With Clustering & Error Pages

Dec 6, 2024 - 7:31 am
Previous Story: Bing Chat Better At News / Sports & Google Bard Can Code, Plus More AI Updates