Hosted On A CDN? You May Get A Special GoogleBot Crawl Rate

Oct 5, 2011 - 8:42 am 8 by

Google Cloud SEOGary Illyes from Google responded to a question in the Google Webmaster Help forums about hosting your content on a CDN, content delivery network.

The question was why did Google assign a special crawl rate to his site.

Gary from Google said, Google may assign content hosted on a CDN with a special crawl rate. He specifically said you should most likely never increase the crawl rate manually if you are on a CDN, at best, leave it for Google to decide. You can lower the crawl rate, but do not increase it.

Gary from Google explained:

When you're using a CDN, we may assign a special crawl rate for your site so that we can crawl as much as possible without affecting negatively others who're using the same service. It's best to not tackle that setting, especially since we're already accessing your site more than 4000 times a day.

If you want to lower your crawling rate, let me know, but I definitely not recommend increasing it. I'd rather focus on posting unique and compelling content and on user experience.

More on CDNs and SEO:

Forum discussion at Google Webmaster Help.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Spam Update, Volatility & Indexing Bug, AI Overviews, Google Maps Exploit, Ads, Applebot - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: June 21, 2024

Jun 21, 2024 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Spam Update, Volatility & Indexing Bug, AI Overviews Tracking & FAQs, Google Maps Exploit, Google Ads, AppleBot & More

Jun 21, 2024 - 8:01 am
Google Updates

Google June 2024 Spam Update Takes Target

Jun 21, 2024 - 7:51 am
Google Search Engine Optimization

Google Search Images Go Missing? Might Be A Quality Issue.

Jun 21, 2024 - 7:41 am

Google Posts Safe Search FAQs

Jun 21, 2024 - 7:31 am

Google Tests Floating Search Bar In Google Discover

Jun 21, 2024 - 7:21 am
Previous Story: My Location Detection Comes To Google Maps Desktop