Hosted On A CDN? You May Get A Special GoogleBot Crawl Rate

Oct 5, 2011 • 8:42 am | comments (8) by twitter Google+ | Filed Under Google Search Engine Optimization

Google Cloud SEOGary Illyes from Google responded to a question in the Google Webmaster Help forums about hosting your content on a CDN, content delivery network.

The question was why did Google assign a special crawl rate to his site.

Gary from Google said, Google may assign content hosted on a CDN with a special crawl rate. He specifically said you should most likely never increase the crawl rate manually if you are on a CDN, at best, leave it for Google to decide. You can lower the crawl rate, but do not increase it.

Gary from Google explained:

When you're using a CDN, we may assign a special crawl rate for your site so that we can crawl as much as possible without affecting negatively others who're using the same service. It's best to not tackle that setting, especially since we're already accessing your site more than 4000 times a day.

If you want to lower your crawling rate, let me know, but I definitely not recommend increasing it. I'd rather focus on posting unique and compelling content and on user experience.

More on CDNs and SEO:

Forum discussion at Google Webmaster Help.

Previous story: My Location Detection Comes To Google Maps Desktop


Simon Temple

10/05/2011 12:52 pm

Explore To is hosted via Amazon, (EC2 via AWS) and have the notice in webmaster tools saying, "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."

larry ganstine

10/05/2011 11:33 pm

I am always looking for ways to boost my traffic.

larry ganstine

10/05/2011 11:35 pm

Always looking for ways to increase my traffic and conversion.


10/06/2011 12:57 pm

Another reason why Cloud services rock!


10/06/2011 12:58 pm

Another reason why Cloud Services rock!


02/13/2012 07:16 pm

Yeah one of my site is also assigned a special crawl rate by Google.

Tushar Bansal

02/06/2013 07:20 am

Is it true that our site crawling is faster then another sites

Mohammad Shenawy

06/30/2013 05:39 am

iam using CDN ,, i need to lower the crawl rate PLZ !!,, my google account is :

blog comments powered by Disqus