GoogleBot Not Sending IF_MODIFIED_SINCE Request?

Oct 9, 2007 • 7:46 am | comments (1) by | Filed Under Google Search Engine Optimization

A WebmasterWorld thread discusses a more detailed issue with how Google's spider, GoogleBot, is crawling some pages. Let me quote the detailed explanation:

I've tried: Checking for the HTTP_IF_MODIFIED_SINCE header and returns "304 Not Modified" if possible.

Problem: Googlebot doesn't always send this header. Even if they already know about a page they doesn't always send the header.

I've tried: Using the expires header to tell google that each page should expire in a month from the request.

Problem: Googlebot keep requesting the pages. They seem to ignore this header.

Brett Tabke, founder of WebmasterWorld, said he noticed these issues as well. jdMorgan, a WebmasterWorld moderator, tried to offer some advice:

Check that the 'expires' header is relative -- Expires after so much time, rather than Expires at a certain time.

You should check your Cache-control server response headers as well.

Is this a Webmaster issue or GoogleBot issue?

Forum discussion at WebmasterWorld.

Previous story: Yahoo Reporting Error: Advertisers Reports Showing Charges in Millions
Ninja Banner
blog comments powered by Disqus