Connection Time & HTTP Status Codes Used By GoogleBot For Crawl Efficiency

Oct 2, 2014 - 8:24 am 8 by

GoogleBotThere is nothing worse than knowing Google is having a tough time accessing your content unintentionally. Well, truth is, if you robots.txt or nofollow your site out of Google by accident, I don't feel bad for you. But if your server flakes out on you, then I do feel your pain.

Yesterday, at SMX East, the great Gary Illyes from the Google search quality group, shared two tidbits that you may not have officially heard on-record from Google about crawl efficiency with GoogleBot.

Now, you know that GoogleBot will play nice with your server. If they feel crawling it too hard will hurt the server, they back off. But what signals do they use for determining that? Google has never really shared that information until yesterday.

They use (1) connection time and (2) server status codes.

If Google sees it takes longer and longer to connect to a web page on your domain between GoogleBots hops, it will figure, it should back off a bit or stop crawling. If GoogleBot is served up HTTP server status codes in the 5xx realm, it will also back off a bit or stop crawling. Of course, it will try again later soon but the last thing Google wants to do is take down your site for users.

So if I were you, I'd have reporting configured on (1) connection time and (2) 5xx server status codes.

Forum discussion at Google+.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Flux, AdSense Ad Intent, California Link Tax & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 23, 2024

Apr 23, 2024 - 4:00 pm
Link Building

Google: Ignore Link Spam Especially To 404 Pages

Apr 23, 2024 - 7:51 am
Google Search Engine Optimization

Google: We Have Taken Action On Some Parasite SEO In Recent Update

Apr 23, 2024 - 7:41 am
Bing Search

Mikhail Parakhin Breaks Silence On Mustafa Suleyman Of Microsoft (Kinda...)

Apr 23, 2024 - 7:31 am
Google Maps

Google Business Profiles Gains Select Preferred Menu Source

Apr 23, 2024 - 7:21 am
Google Search Engine Optimization

Google: Crawl Budget Goes Across All Googlebot Crawling, Not Just Web Search

Apr 23, 2024 - 7:11 am
Previous Story: Google Expands Dynamic Remarketing Ads