Google Fetch As Googlebot Won't Crawl When Site Is Too Slow

Dec 23, 2013 - 8:30 am 2 by

Google Page SpeedA Google Webmaster Help thread has one webmaster complaining his site isn't being crawled by Google and isn't showing up in the search results. The reason, his site can't handle Googlebot crawling it.

The site is pretty static and basic but the server is a cheap or free host that can't handle much activity. So Googlebot can't crawl it without taking down the site and thus stays away until it can get through to crawl it without negatively impacting the site.

The interesting thing is that if you use the Fetch As Googlebot feature when this is the case, it will fail as well. So you can actually somewhat diagnose a major site speed issue with Fetch as Googlebot.

John Mueller from Google said:

Looking at your site, I do see that we'd like to crawl more from the server, but we're holding back because we think the server might not be able to handle the load. This is the reason why the Fetch as Google requests aren't making it through. In particular, we're seeing a fairly high response-time for URLs from the server, which often signals that the server is pretty busy even without us crawling.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Updates

Google Search Ranking Volatility This Weekend - Around July 18th

Jul 19, 2026 - 8:04 am
Search Forum Recap

Daily Search Forum Recap: July 17, 2026

Jul 17, 2026 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google 7/11 Update, Bing Penalties, Google Images, AI Overviews AI Images, Google Ads Updates, ChatGPT Ads Features & Apple Maps

Jul 17, 2026 - 8:01 am
Google Ads

Google Ads Making Broader Smart Bidding Updates? Google Says No.

Jul 17, 2026 - 7:51 am
Google Maps

Did Google Maps Turn Off OpenTable Reservations?

Jul 17, 2026 - 7:41 am
Google Search Engine Optimization

Google Renamed NotebookLM Useragent To Google-GeminiNotebook

Jul 17, 2026 - 7:31 am
 
Previous Story: Google's Happy Holidays Logos Start Going Live