Why Google May Not Find Your Sitemap File

Oct 11, 2010 - 8:53 am 4 by

I'll be honest, I am pretty technical, but I simply do not fully understand the issue here.

Franz Enzenhofer posted a thread at Google Webmaster Help claiming Google cannot access his XML Sitemap no matter what he does. After much debugging and testing, he wrote a blog post explaining why he and maybe your servers won't serve up your XML sitemap file to Google.

He wasn't able to see a GET request by Google, so he dug even deeper and noticed an issue at the TCP/IP level. He said, and I quote:

  • You realize that googlebot makes multiple (we counted up to 11) GET requests in one single TCP/IP connection. (which is OK according to the HTTTP 1.1 spec).
  • You realize (with the help of stackoverflow) that these multiple GET requests in the some TCP/IP connection are processed in sequence (one after the other).
  • You realize that if one these GET requests has a major time lag (is much slower than the other GET requests) Google cuts the TCP/IP connection.
  • Because all the GET requests in the connection were processed in sequence, all the GET requests after the cut are lost. You don’t see them in the error/access logs as they were never processed, even though they were sent.
  • You see an error in Google Webmaster Tools, without a trace in your logfiles.

Google's Matt Cutts commented on the blog post saying, "Interesting--good find." He didn't necessarily confirm this is a Google bug but JohnMu commented in the Google Webmaster Help thread implying that GoogleBot is stopping after some time due to his server speed? At least I think that is what he is saying:

It looks like you were busy while I was out of town :). Yes, this error can mean that we did not even try to fetch the Sitemap file from your server. If we recently picked up some of the Sitemap files again, I assume this is likely just a coincidence. With your site, we'd likely need to crawl with more than 10 QPS (the maximum manual setting in the crawl rate tool); by changing it to "Let Google determine my crawl rate (recommended)" our systems would be able to do that if they're able to determine that your server can handle it.

Forum discussion at Google Webmaster Help.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Weekend Volatility, Google On Search Leak, Elizabeth Tucker Interview & Apple Intelligence - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: June 19, 2024

Jun 19, 2024 - 10:00 am

Report On If Google Showing Fewer Reddit Links In Search?

Jun 19, 2024 - 7:51 am

Google AI Overviews FAQs Including Why You Can't Disable AI Overviews

Jun 19, 2024 - 7:41 am
Google Search Engine Optimization

Google: Two Common Reasons When A Spike In Crawling Is Bad

Jun 19, 2024 - 7:31 am

Google Tests Two More People Also Ask Results (6 PPA)

Jun 19, 2024 - 7:21 am
Google Maps

Google Local Panels Gains Menu Button

Jun 19, 2024 - 7:11 am
Previous Story: Don't Use Pipes In Your URLS | Says Google