(0)
Google's John Mueller said on Twitter that Google "never supported crawl-delay in robots.txt." Bing's Frédéric Dubut said on Twitter "Bingbot supports crawl-delay." That is not to say Google does not let you slow the crawl, you can and here is how.
Here are the tweets:
We never supported crawl-delay in robots.txt -- the numbers given there tend to be unrealistic for the most part.
— 🍌 John 🍌 (@JohnMu) December 12, 2019
Yes Bingbot supports crawl-delay.
— Frédéric Dubut (@CoperniX) December 12, 2019
The crawl rate setting is still in the legacy Search Console, but you can still access it over here. You can learn more about Google's crawl rate over here.
Forum discussion at Twitter.
