DuplexWeb-Google - GoogleBot Crawler For Duplex / Google Assistant

Jun 3, 2021 • 7:11 am | comments (0) by twitter | Filed Under Google Search Engine Optimization
 

Google added a support document for the web crawler Google uses for Duplex, its Google Assistant voice features that can have conversations with people. The bot's useragent is named DuplexWeb-Google and is now part of the modern set of GoogleBot crawlers.

Google wrote that "DuplexWeb-Google is the user agent that supports the Duplex on the web service."

Here is how it crawls:

  • No services that use the DuplexWeb-Google user agent will perform any purchases or perform any other significant actions when crawling your site.
  • The DuplexWeb-Google user agent crawls occur a few times a day to a few times an hour, depending on the feature being trained, but these runs are calculated to not overload your site or disturb your traffic.
  • The DuplexWeb-Google user agent crawls are not used by Google Search for indexing. Because they are not used for indexing, the DuplexWeb-Google user agent does not recognize the noindex directive.
  • Google Analytics does not record page requests made by the DuplexWeb-Google user agent during crawling and analysis.

Google said to block it, "you must explicitly block the DuplexWeb-Google user agent using the Disallow robots.txt rule to prevent it from crawling your site." DuplexWeb bot will follow the robots.txt ruleset with the exception of:

  • When Duplex on the web is enabled using Search Console (the default), the DuplexWeb-Google user agent ignores the Disallow rules in the * wildcard user agent groups.
  • When Duplex on the web is disabled using Search Console, the DuplexWeb-Google user agent respects Disallow rules in the * wildcard user agent groups.

Just to be clear, Duplex is not new, it has been around since 2018 or earlier. But I never saw any details on a spider/bot for Duplex before, I am sure it has been out there but I never saw it.

Forum discussion at Twitter.

Previous story: Daily Search Forum Recap: June 2, 2021
 
blog comments powered by Disqus