I'll be honest, I am pretty technical, but I simply do not fully understand the issue here.
Franz Enzenhofer posted a thread at Google Webmaster Help claiming Google cannot access his XML Sitemap no matter what he does. After much debugging and testing, he wrote a blog post explaining why he and maybe your servers won't serve up your XML sitemap file to Google.
He wasn't able to see a GET request by Google, so he dug even deeper and noticed an issue at the TCP/IP level. He said, and I quote:
- You realize that googlebot makes multiple (we counted up to 11) GET requests in one single TCP/IP connection. (which is OK according to the HTTTP 1.1 spec).
- You realize (with the help of stackoverflow) that these multiple GET requests in the some TCP/IP connection are processed in sequence (one after the other).
- You realize that if one these GET requests has a major time lag (is much slower than the other GET requests) Google cuts the TCP/IP connection.
- Because all the GET requests in the connection were processed in sequence, all the GET requests after the cut are lost. You don’t see them in the error/access logs as they were never processed, even though they were sent.
- You see an error in Google Webmaster Tools, without a trace in your logfiles.
Google's Matt Cutts commented on the blog post saying, "Interesting--good find." He didn't necessarily confirm this is a Google bug but JohnMu commented in the Google Webmaster Help thread implying that GoogleBot is stopping after some time due to his server speed? At least I think that is what he is saying:
It looks like you were busy while I was out of town :). Yes, this error can mean that we did not even try to fetch the Sitemap file from your server. If we recently picked up some of the Sitemap files again, I assume this is likely just a coincidence. With your site, we'd likely need to crawl with more than 10 QPS (the maximum manual setting in the crawl rate tool); by changing it to "Let Google determine my crawl rate (recommended)" our systems would be able to do that if they're able to determine that your server can handle it.
Forum discussion at Google Webmaster Help.
