DuplexWeb-Google - GoogleBot Crawler For Duplex / Google Assistant

Jun 3, 2021 - 7:11 am 0 by

Google Robot Duplex

Google added a support document for the web crawler Google uses for Duplex, its Google Assistant voice features that can have conversations with people. The bot's useragent is named DuplexWeb-Google and is now part of the modern set of GoogleBot crawlers.

Google wrote that "DuplexWeb-Google is the user agent that supports the Duplex on the web service."

Here is how it crawls:

  • No services that use the DuplexWeb-Google user agent will perform any purchases or perform any other significant actions when crawling your site.
  • The DuplexWeb-Google user agent crawls occur a few times a day to a few times an hour, depending on the feature being trained, but these runs are calculated to not overload your site or disturb your traffic.
  • The DuplexWeb-Google user agent crawls are not used by Google Search for indexing. Because they are not used for indexing, the DuplexWeb-Google user agent does not recognize the noindex directive.
  • Google Analytics does not record page requests made by the DuplexWeb-Google user agent during crawling and analysis.

Google said to block it, "you must explicitly block the DuplexWeb-Google user agent using the Disallow robots.txt rule to prevent it from crawling your site." DuplexWeb bot will follow the robots.txt ruleset with the exception of:

  • When Duplex on the web is enabled using Search Console (the default), the DuplexWeb-Google user agent ignores the Disallow rules in the * wildcard user agent groups.
  • When Duplex on the web is disabled using Search Console, the DuplexWeb-Google user agent respects Disallow rules in the * wildcard user agent groups.

Just to be clear, Duplex is not new, it has been around since 2018 or earlier. But I never saw any details on a spider/bot for Duplex before, I am sure it has been out there but I never saw it.

Forum discussion at Twitter.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Volatility, Indexing Bugs, Google Ads Broad Match Default & YouTube SEO Tips - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 18, 2024

Jul 18, 2024 - 10:00 am
Google Updates

Google Search Ranking Algorithm Volatility - July 18th

Jul 18, 2024 - 7:51 am

Google Tests AI Overviews In Workspace Accounts In The UK

Jul 18, 2024 - 7:41 am

Google Bolding Query In Search Box When Cursor Out Of Search Box

Jul 18, 2024 - 7:31 am
Google Search Engine Optimization

Google Merchant Center Next Coming To All Merchants

Jul 18, 2024 - 7:21 am
Google Search Engine Optimization

Google's Site Reputation Abuse Update Leads Gaming Affiliate To Reduce Financial Forecast

Jul 18, 2024 - 7:11 am
Previous Story: Web Structure In Google Lobby