Why Google May Not Find Your Sitemap File

Oct 11, 2010 - 8:53 am 4 by

I'll be honest, I am pretty technical, but I simply do not fully understand the issue here.

Franz Enzenhofer posted a thread at Google Webmaster Help claiming Google cannot access his XML Sitemap no matter what he does. After much debugging and testing, he wrote a blog post explaining why he and maybe your servers won't serve up your XML sitemap file to Google.

He wasn't able to see a GET request by Google, so he dug even deeper and noticed an issue at the TCP/IP level. He said, and I quote:

  • You realize that googlebot makes multiple (we counted up to 11) GET requests in one single TCP/IP connection. (which is OK according to the HTTTP 1.1 spec).
  • You realize (with the help of stackoverflow) that these multiple GET requests in the some TCP/IP connection are processed in sequence (one after the other).
  • You realize that if one these GET requests has a major time lag (is much slower than the other GET requests) Google cuts the TCP/IP connection.
  • Because all the GET requests in the connection were processed in sequence, all the GET requests after the cut are lost. You don’t see them in the error/access logs as they were never processed, even though they were sent.
  • You see an error in Google Webmaster Tools, without a trace in your logfiles.

Google's Matt Cutts commented on the blog post saying, "Interesting--good find." He didn't necessarily confirm this is a Google bug but JohnMu commented in the Google Webmaster Help thread implying that GoogleBot is stopping after some time due to his server speed? At least I think that is what he is saying:

It looks like you were busy while I was out of town :). Yes, this error can mean that we did not even try to fetch the Sitemap file from your server. If we recently picked up some of the Sitemap files again, I assume this is likely just a coincidence. With your site, we'd likely need to crawl with more than 10 QPS (the maximum manual setting in the crawl rate tool); by changing it to "Let Google determine my crawl rate (recommended)" our systems would be able to do that if they're able to determine that your server can handle it.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Updates

Google Urges Patience As The March 2024 Core Update Continues To Rollout

Mar 18, 2024 - 7:51 am
Google

Official: Google Replaces Perspective Filter With Forums Filter

Mar 18, 2024 - 7:41 am
Google Maps

Google Business Profiles Now Offers Additional Review After Appeal Is Denied

Mar 18, 2024 - 7:31 am
Google Maps

EU Searchers Complaining About Google Maps Features Changes Related To DMA

Mar 18, 2024 - 7:21 am
Google

Google Showing Fewer Sitelinks Within Search

Mar 18, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: March 15, 2024

Mar 15, 2024 - 4:00 pm
Previous Story: Don't Use Pipes In Your URLS | Says Google