Google: Crawling At Large Scale Is Complicated; Don't Oversimplify It

Jan 8, 2020 - 7:52 am 0 by

Google Complicated

Google's John Mueller responded to a question related to crawl budget around large scale web sites saying "crawling at a large scale is much more complicated." This was when he proposed two ideas for improving crawl budget but John "both of those are too simplified" and there is more to it.

Here is the post on Twitter:

Gary Illyes from Google did do a deep post on crawl budget for those that are interested, this was a few years ago. But Google has been saying for most sites its overrated to focus on it.

In either case - crawl budget cannot be a simple thing for really large sites.

Forum discussion at Twitter.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Spam Update, Volatility & Indexing Bug, AI Overviews, Google Maps Exploit, Ads, Applebot - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: June 24, 2024

Jun 24, 2024 - 10:00 am
Google Ads

Google Ads To Bring Automatically Created Assets To Ad Level?

Jun 24, 2024 - 7:51 am

Google Perspectives Can Show Instagram Posts

Jun 24, 2024 - 7:41 am
Google Search Engine Optimization

Google Structured Data Carousel Guidance On How To Markup Multiple Items On Category Pages

Jun 24, 2024 - 7:31 am
Google Maps

Google Using AI Overviews On Business Profiles Local Listings

Jun 24, 2024 - 7:21 am
Google Search Engine Optimization

Google: When You Should Fix Pages 404 Status Codes

Jun 24, 2024 - 7:11 am
Previous Story: Google User Comments In Search Tests Expanding To TV Shows