Is Google Crawling & Indexing All of My Pages?
Michael Gray has composed a post that helps SEOs find out which pages of their site haven't been crawled, which becomes increasingly more important due to Google's removal of the supplemental index. He explains that you should put a timestamp on every single page, after which you wait about 2 months. At that point, you can perform a search for the article plus the date and you should get your results.
Michael then explains how you can optimize those pages to have them become crawled and provides other informative insights as well.
Is it worth the hassle? It depends on how important this is to you. The goal should be to integrate code into the page so that you don't have to manually add timestamps to every single page. Still, if it is important to you and you want to know (but don't have the resources available to code it in the footer), it may be worth the effort to do it.
Forum discussion continues at Sphinn.
Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!
Tamar Weinberg in Google Optimization at September 11, 2008 9:23 AM
Comments (1)
Sphinn 
• Tweet This! 
• Digg 
• Delicious 
• StumbleUpon 
• Google Co-op 
Comments
Unfortunately, his proposed methodology is completely bogus, since he is operating under the false assumption that Google did away with its Supplemental Results Index.
They never said they were doing away with it. They only said they were hiding it from people.
In fact, the Supplemental Results Index is still there and Google still does not fully parse and index the words on pages in the Supplemental Results Index.
Hence, Google WILL crawl and "index" pages very frequently, but because they get stuck in the Supplemental Results Index those pages are less likely to appear for Michael's date stamp search.
There are certainly reasons to include date stamps on your pages (not all search engines provide you with a caching date), but this plan just won't accomplish anything useful for the SEO industry.
Posted by Michael Martinez at September 11, 2008 12:15