Why Google May Not Find Your Sitemap File

Oct 11, 2010 - 7:53 am 4 by

I'll be honest, I am pretty technical, but I simply do not fully understand the issue here.

Franz Enzenhofer posted a thread at Google Webmaster Help claiming Google cannot access his XML Sitemap no matter what he does. After much debugging and testing, he wrote a blog post explaining why he and maybe your servers won't serve up your XML sitemap file to Google.

He wasn't able to see a GET request by Google, so he dug even deeper and noticed an issue at the TCP/IP level. He said, and I quote:

  • You realize that googlebot makes multiple (we counted up to 11) GET requests in one single TCP/IP connection. (which is OK according to the HTTTP 1.1 spec).
  • You realize (with the help of stackoverflow) that these multiple GET requests in the some TCP/IP connection are processed in sequence (one after the other).
  • You realize that if one these GET requests has a major time lag (is much slower than the other GET requests) Google cuts the TCP/IP connection.
  • Because all the GET requests in the connection were processed in sequence, all the GET requests after the cut are lost. You don’t see them in the error/access logs as they were never processed, even though they were sent.
  • You see an error in Google Webmaster Tools, without a trace in your logfiles.

Google's Matt Cutts commented on the blog post saying, "Interesting--good find." He didn't necessarily confirm this is a Google bug but JohnMu commented in the Google Webmaster Help thread implying that GoogleBot is stopping after some time due to his server speed? At least I think that is what he is saying:

It looks like you were busy while I was out of town :). Yes, this error can mean that we did not even try to fetch the Sitemap file from your server. If we recently picked up some of the Sitemap files again, I assume this is likely just a coincidence. With your site, we'd likely need to crawl with more than 10 QPS (the maximum manual setting in the crawl rate tool); by changing it to "Let Google determine my crawl rate (recommended)" our systems would be able to do that if they're able to determine that your server can handle it.

Forum discussion at Google Webmaster Help.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Search Ranking Update, Recipe Blogs Drop, Google Hits Reviews & More SEO, PPC and Local - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google: Sites Wrongfully Using The Google Indexing API Often Are Spammy & Low Quality

Feb 20, 2024 - 7:51 am
Google Search Engine Optimization

Google Merchant Center Updates Estimated Delivery Times Calculation

Feb 20, 2024 - 7:41 am
Google Search Engine Optimization

Google Merchant Center Requires Meta Labels On AI Generated Images

Feb 20, 2024 - 7:31 am
Google Maps

Google Local Pack Tests Restaurant Price Ranges

Feb 20, 2024 - 7:21 am

Google Search Can Wait On Customer Service Hold For You

Feb 20, 2024 - 7:11 am
Google Search Engine Optimization

New Google Search Support For Product Variant Structured Data

Feb 20, 2024 - 6:00 am
Previous Story: Don't Use Pipes In Your URLS | Says Google