Gary Illyes and Chery Prommawin released the final video in the Non-Canonical Cabinet series, this one is on CDNs, content delivery networks. They discuss why migrating a website to a CDN can suddenly increase Googlebot crawling activity, why this may overload your origin server, and how to avoid accidentally blocking Google's crawlers.
Google also posted a new blog post on this topic.
Here is the video, it is less than 10 minutes long:
In this episode, they explain "why Googlebot is firehosing your newly set up CDN, what to watch out for, and how to handle the situation calmly" including:
- Crawler Over-enthusiasm: Why crawlers ramp up crawl speed when you move to a CDN
- Cache Misses & Origin Servers: Why for every unique first-time request, your origin server still does the heavy lifting anyway
- Blanket Bot Protection: Does aggressively blocking all bots help?
- How to Survive CDN Launch Day: Things you can do to protect your servers from collapsing
While CDNs are great, migrating to one can sometimes create unexpected problems, particularly with search engine crawlers.
Googlebot automatically monitors how quickly a website responds to its requests. When a website moves to a CDN, response times can become significantly faster. As a result, Googlebot may interpret this improved performance as an opportunity to crawl more pages at a higher rate. So, your website might suddenly receive significantly more crawling requests than before.
When you first deploy a CDN, its cache is initially empty. This means it doesn't have copies of your website's content yet. Imagine Googlebot starts crawling 100,000 unique URLs. If those pages aren't cached yet, the CDN must retrieve each page from your origin server. So even if you've implemented a CDN, your origin server can still see a massive traffic spike during the initial crawl. A CDN does not immediately protect your origin server from every request. Its effectiveness depends partly on how much content is already cached.
When developers notice a sudden spike in crawling activity, they may assume their website is being attacked by malicious bots. In response, they might enable aggressive bot protection through their CDN. The problem is that these protections can accidentally block legitimate search engine crawlers, including Googlebot. Blocking legitimate search engine crawlers can create a much bigger SEO problem than the original increase in crawling activity.
A good approach when migrating to a CDN is:
- Prepare and pre-warm your CDN cache.
- Allow verified search engine crawlers.
- Block malicious bots selectively.
- Use rate limiting to manage excessive traffic.
- Monitor server health and crawling activity.
Most, if not all, of this content comes from Google's help documents including:
- Moving to a CDN without overwhelming your origin (or blocking Googlebot)
- Reduce the Google crawl rate
- Optimize your crawl budget
Forum discussion at LinkedIn.
