When Google's Algorithms Don't Index Your Content

Jan 14, 2013 - 9:08 am 24 by

Google Index AlgorithmA Google Webmaster Help thread has one webmaster upset that his index count, compared to the number of pages submitted via his sitemap file is continuously going down, as opposed to up.

One of the best ways to see how many pages Google has indexed of your web site is to upload an XML Sitemap file and compare the URLs submitted to the URLs indexed. If that count is close, that is a good thing. If the number of URLs indexed continues to go up, that is a good thing. If that number continuously goes down, there is likely a problem.

So this webmaster wanted to know what the problem was and Gary Illyes from Google explained that Google's algorithms don't want to index many of the pages. He wrote:

As we improve our algorithms, they may decide to not reindex pages that are likely to be not useful for the users. I took a look on the pages that were once indexed but currently aren't and it appears there are quite a few that have no real content.

He showed examples of pages that are soft 404s (it says page not found but returns a 200 status code in the http header). He also showed examples of blank pages being index. Also, he showed examples of URLs in the Sitemap that is referencing URLs that are not canonical.

Healthy sites, need health URLs, content, redirects and proper http header responses. Otherwise, Google may stop indexing and worse, crawling the URLs.

Forum discussion at Google Webmaster Help.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: October 3, 2024

Oct 3, 2024 - 10:00 am
Google Updates

Record-Breaking Google Search Ranking Volatility Continues Into October

Oct 3, 2024 - 8:01 am
Bing Ads

Microsoft Advertising New Ad Experience, Diagnostics & Performance Snapshot

Oct 3, 2024 - 7:51 am
Google

Google Testing Search Instead For Section

Oct 3, 2024 - 7:41 am
Google

Google: SEOs Can Help Shape Client Policies & Decisions On AI Bots

Oct 3, 2024 - 7:31 am
Google Ads

Google Ads With Images & Map Carousel

Oct 3, 2024 - 7:21 am
Previous Story: Google AdSense Testing No Arrows Now?