Xenu hammers servers and people with large contenht sites should be VERY reluctant to turn it loose on their pages.

Google Webmaster Central has been reporting a lot of bogus 404 issues lately. I suspect server timeouts may be one possible cause as a server I have been monitoring is getting more of these bogus 404 errors when its traffic increases.

I'm not too happy with the quality of service coming from Google Webmaster Tools right now. It looks to me like they need to put more resources into the mix.

Many of the 404 can be out of our control -- such as when other sites link to you and misprint your URL. Google WMC only reports the page that cannot be found, but not where they were being linked from -- a feature that I hope they will implement in the future.

There will always be 404, whether your pages no longer exist, users type it in wrong, bad links from internal and external sites, etc.

I don't necessarily think 404 it's a bad thing -- unless, of course, the bad links are internal. In fact, I think 404 page is a great opportunity to create links to other content of your site -- almost like another sitemap -- driving more spider traffic and help frustrated users get to where they want to go.

I've implemented something at Kelley Blue Book recently. Just type in www.kbb.com/anyrandomfile to see.