Google has added two new Googlebots, Google crawlers, to its crawler list. They include GoogleOther-Image and GoogleOther-Video, which are part of the GoogleOther category of bots and are used for crawling of binary data for research and development reasons.
Google wrote, "Added two new crawlers, GoogleOther-Image and GoogleOther-Video, which are versions of GoogleOther optimized for fetching image and video bytes respectively. While at it, we also updated the list of user agent strings of GoogleOther to better reflect the most active user agent versions."
"The new crawlers were launched to better support crawling of binary data that may be used for research and development," Google added.
In April 2023, Google added GoogleOther as a crawler, which was a "generic crawler that may be used by various product teams for fetching publicly accessible content from sites." Gary Illyes from Google explained back then that the crawler will "replace some of Googlebot's other jobs like R&D crawls to free up some crawl capacity for Googlebot."
He added, these bots are "used by Google product teams for internally building the Google index." "Googlebot is used for index building, GoogleOther for the other tasks historically owned by Googlebot," he added.
These bots can be found in the documentation here:
- GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs. It goes under the User agent tokens; GoogleOther-Image and GoogleOther and uses the full user agent string GoogleOther-Image/1.0.
- GoogleOther-Video is the version of GoogleOther optimized for fetching publicly accessible video URLs. It goes under the User agent tokens of GoogleOther-Video and GoogleOther and uses the full user agent string GoogleOther-Video/1.0.
Here is a screenshot of the three GoogleOther crawlers:
Is GoogleOther crawling too much?
You can also use the crawling feedback form linked at https://t.co/edeLqlDY51 if there's something specific to pass on - this is read by the folks running the Google crawlers.
— John 🧀 ... 🧀 (@JohnMu) May 17, 2024
Forum discussion at X.
