Google: Crawling At Large Scale Is Complicated; Don't Oversimplify It

Jan 8, 2020 - 7:52 am 0 by

Google Complicated

Google's John Mueller responded to a question related to crawl budget around large scale web sites saying "crawling at a large scale is much more complicated." This was when he proposed two ideas for improving crawl budget but John "both of those are too simplified" and there is more to it.

Here is the post on Twitter:

Gary Illyes from Google did do a deep post on crawl budget for those that are interested, this was a few years ago. But Google has been saying for most sites its overrated to focus on it.

In either case - crawl budget cannot be a simple thing for really large sites.

Forum discussion at Twitter.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: May 28, 2024

May 28, 2024 - 10:00 am
Google Updates

Memorial Day Google Search Ranking Volatility

May 28, 2024 - 7:51 am
Google Search Engine Optimization

Google's John Mueller On Recovering From Core Updates - Maybe You Had A Good Run...

May 28, 2024 - 7:41 am
Google Ads

Undated Google Ads Experiments To End August 23, 2024

May 28, 2024 - 7:31 am

Google Tests Thin Top Deals Search Bar

May 28, 2024 - 7:21 am
Google Search Engine Optimization

Report: 14,000+ Google Search Ranking Features Leaked

May 28, 2024 - 6:15 am
Previous Story: Google User Comments In Search Tests Expanding To TV Shows