DuplexWeb-Google - GoogleBot Crawler For Duplex / Google Assistant

Jun 3, 2021 - 7:11 am 0 by

Google Robot Duplex

Google added a support document for the web crawler Google uses for Duplex, its Google Assistant voice features that can have conversations with people. The bot's useragent is named DuplexWeb-Google and is now part of the modern set of GoogleBot crawlers.

Google wrote that "DuplexWeb-Google is the user agent that supports the Duplex on the web service."

Here is how it crawls:

  • No services that use the DuplexWeb-Google user agent will perform any purchases or perform any other significant actions when crawling your site.
  • The DuplexWeb-Google user agent crawls occur a few times a day to a few times an hour, depending on the feature being trained, but these runs are calculated to not overload your site or disturb your traffic.
  • The DuplexWeb-Google user agent crawls are not used by Google Search for indexing. Because they are not used for indexing, the DuplexWeb-Google user agent does not recognize the noindex directive.
  • Google Analytics does not record page requests made by the DuplexWeb-Google user agent during crawling and analysis.

Google said to block it, "you must explicitly block the DuplexWeb-Google user agent using the Disallow robots.txt rule to prevent it from crawling your site." DuplexWeb bot will follow the robots.txt ruleset with the exception of:

  • When Duplex on the web is enabled using Search Console (the default), the DuplexWeb-Google user agent ignores the Disallow rules in the * wildcard user agent groups.
  • When Duplex on the web is disabled using Search Console, the DuplexWeb-Google user agent respects Disallow rules in the * wildcard user agent groups.

Just to be clear, Duplex is not new, it has been around since 2018 or earlier. But I never saw any details on a spider/bot for Duplex before, I am sure it has been out there but I never saw it.

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: December 12, 2024

Dec 12, 2024 - 10:00 am
Google

Google Is Testing Gemini 2.0 Powered AI Overviews In Search

Dec 12, 2024 - 7:51 am
Google Ads

Google Ads Testing Video Assets For Search Campaigns?

Dec 12, 2024 - 7:41 am
Apple App Store

Apple Visual Intelligence Is Out With ChatGPT & Google Integration

Dec 12, 2024 - 7:31 am
Bing Search

Official: Bing Search Drops Cache Link

Dec 12, 2024 - 7:21 am
Google

Google Tests Sitelinks With Arrow Down To Show More

Dec 12, 2024 - 7:11 am
Previous Story: Web Structure In Google Lobby