Google: Crawling At Large Scale Is Complicated; Don't Oversimplify It

Jan 8, 2020 - 7:52 am 0 by

Google Complicated

Google's John Mueller responded to a question related to crawl budget around large scale web sites saying "crawling at a large scale is much more complicated." This was when he proposed two ideas for improving crawl budget but John "both of those are too simplified" and there is more to it.

Here is the post on Twitter:

Gary Illyes from Google did do a deep post on crawl budget for those that are interested, this was a few years ago. But Google has been saying for most sites its overrated to focus on it.

In either case - crawl budget cannot be a simple thing for really large sites.

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 15, 2024

Mar 15, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More

Mar 15, 2024 - 8:01 am
Google Updates

Google March 2024 Core & Spam Update Movement Today

Mar 15, 2024 - 7:51 am
Google

Google Image Search Results Testing Like Button

Mar 15, 2024 - 7:41 am
Google

Google Merchant Center Product Studio With Themed Templates Including St. Patrick's Day

Mar 15, 2024 - 7:31 am
Google Maps

Google Search Dishes Nearby Carousel

Mar 15, 2024 - 7:21 am
Previous Story: Google User Comments In Search Tests Expanding To TV Shows