(3)
A WebmasterWorld thread asks how long does Google send out their crawlers to a page that has been removed (i.e. a 404 error not found status)?
The simple answer, as mentioned in the thread is, forever.
Google will continue to try to see if the 404ed page has ever been reinstated as a live page.
Google even provides these errors in Google Webmaster Central, where you can see web crawl errors including 404 not found errors. (note to Vanessa, fix the link on here for the 404 link to go to here).
Why does Google continue to try to access these pages? As g1smd said:
Say you bought a domain name from someone and put up a new site. Say that Google refused to pick up your /about.html and /contact.html pages, and that eventually you found that it was all because once a page went 404 Google refused to ever look at that URL again.You would think that was a bad policy. That is why it doesn't work like that.
Forum discussion at WebmasterWorld.
Comments:
Icheb
03/09/2007 02:39 pm
There's potential for a misunderstanding here. What you probably mean is pages that still have links pointing to them but don't exist at the moment of the crawl. The other type would be for example when people change the site architecture and all addresses change at the same time. When there are no deep links to those pages from other domains, it wouldn't make sense to crawl those pages.
Cvos SEO
03/09/2007 11:08 pm
Google will definitely attempt to crawl and list missing pages. I have one site where a page for a fairly competitive search term that was removed in 2002 STILL sometimes appears in a search result.
Vanessa Fox
03/10/2007 12:20 am
Thanks Barry. That link should now be fixed.