Did Google Stop Crawling Blogger Blogs on Personalized Domains?

Aug 7, 2008 - 7:26 am 13 by

There are threads at Google Groups and DigitalPoint Forums with multiple reports of Google not crawling Blogger hosted blogs, that are on custom or private domains (i.e. not on blogspot.com domains). Many have reported that the Googlebot crawling has stopped as of July 30th.

I asked Branko to help me see any pattern to the issue and we found a couple things. Both domains listed in the thread have similar patterns.

(1) They both have a robots.txt file that blocks a Google crawer, Mediapartners-Google. Now, we know that Google uses multiple crawlers to index content and they all work together. Branko theorized that maybe that "reduced frequency due to reducing the number of spiders that visit his site." I agree, that is a possibility.

(2) This is more striking to us, is that if you view the source code of both of the reported blogs, you will notice a widget from a script named SetSystemMarkup. I am not sure what that widget does, but in the widget, it has NOINDEX,NOFOLLOW in it. Maybe Google is misinterpreting that as a META tag? I do see that these blogs are both in the Google index, but maybe this is impacting any recent crawl activity for these sites.

I do not think this is a Blogger or Blogspot.com wide issue.

Forum discussion at Google Groups and DigitalPoint Forums.


Popular Categories

The Pulse of the search community


Search Video Recaps

Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Video Recaps

Search News Buzz Video Recap: Google Helpful Content System, Reddit’s Google Deal, Product Variant Schema, Google Ads, Bing Search & Search Volume Predictions

Feb 23, 2024 - 8:01 am
Google Ads

Google Ad Strength For PMAX To Weigh Assets Quantity & Variety More As AI Expands

Feb 23, 2024 - 7:51 am
Google Search Engine Optimization

Google: You Decide What Old Content Is Helpful Or Not

Feb 23, 2024 - 7:41 am

No Joke: Google Is Paying Reddit For More Content, More Often

Feb 23, 2024 - 7:31 am
Google Ads

Google Ads API Version 16 Now Available

Feb 23, 2024 - 7:21 am
Bing Search

Bing Search Yellow Opinion Label On News Stories

Feb 23, 2024 - 7:11 am
Previous Story: Google Fixes Search Volume Bug in AdWords Keyword Tool