Google Does Not Support Crawl Delay But Bing Does

Dec 13, 2019 - 7:43 am 0 by

Crawl Delay

Google's John Mueller said on Twitter that Google "never supported crawl-delay in robots.txt." Bing's Frédéric Dubut said on Twitter "Bingbot supports crawl-delay." That is not to say Google does not let you slow the crawl, you can and here is how.

Here are the tweets:

The crawl rate setting is still in the legacy Search Console, but you can still access it over here. You can learn more about Google's crawl rate over here.

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Spam Update, Volatility & Indexing Bug, AI Overviews, Google Maps Exploit, Ads, Applebot - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: June 24, 2024

Jun 24, 2024 - 10:00 am
Google Ads

Google Ads To Bring Automatically Created Assets To Ad Level?

Jun 24, 2024 - 7:51 am
Google

Google Perspectives Can Show Instagram Posts

Jun 24, 2024 - 7:41 am
Google Search Engine Optimization

Google Structured Data Carousel Guidance On How To Markup Multiple Items On Category Pages

Jun 24, 2024 - 7:31 am
Google Maps

Google Using AI Overviews On Business Profiles Local Listings

Jun 24, 2024 - 7:21 am
Google Search Engine Optimization

Google: When You Should Fix Pages 404 Status Codes

Jun 24, 2024 - 7:11 am
Previous Story: Google: Speakable Markup Works Outside Of News Content