Did Google Stop Crawling Blogger Blogs on Personalized Domains?

Aug 7, 2008 • 8:26 am | comments (13) by twitter Google+ | Filed Under Google Search Engine Optimization
 

There are threads at Google Groups and DigitalPoint Forums with multiple reports of Google not crawling Blogger hosted blogs, that are on custom or private domains (i.e. not on blogspot.com domains). Many have reported that the Googlebot crawling has stopped as of July 30th.

I asked Branko to help me see any pattern to the issue and we found a couple things. Both domains listed in the thread have similar patterns.

(1) They both have a robots.txt file that blocks a Google crawer, Mediapartners-Google. Now, we know that Google uses multiple crawlers to index content and they all work together. Branko theorized that maybe that "reduced frequency due to reducing the number of spiders that visit his site." I agree, that is a possibility.

(2) This is more striking to us, is that if you view the source code of both of the reported blogs, you will notice a widget from a script named SetSystemMarkup. I am not sure what that widget does, but in the widget, it has NOINDEX,NOFOLLOW in it. Maybe Google is misinterpreting that as a META tag? I do see that these blogs are both in the Google index, but maybe this is impacting any recent crawl activity for these sites.

I do not think this is a Blogger or Blogspot.com wide issue.

Forum discussion at Google Groups and DigitalPoint Forums.

Previous story: Google Fixes Search Volume Bug in AdWords Keyword Tool
 

Comments:

Taylor

08/07/2008 05:34 pm

So man, do you know how we bloggers can fix the issue?

Sunil

08/08/2008 04:19 am

I did not find anything like SetSystemMarkup in my blog, also robot.txt is fine. There is something else that's happening.

Sunil

08/08/2008 04:22 am

Oh Man, This is my robots.txt file User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Sitemap: http://www.dailytechnologytips.com/feeds/posts/default?orderby=updated WTF is this?

Branko

08/08/2008 06:19 am

Sunil check the source code of your homepage. if you search for "noindex" you will find that the line where it appears, belongs to a javascript called 1980098066-widgets.js. Now I don't know whether this is the reason for your site not being crawled as frequently as before but it is suspicious. I would remove that widget somehow and see if the situation improves. And remove the blocking of Mediapartners-Google robot and see if that works for you. It would also be nice of you to update here or on the Google group on the outcome of this changes.

Sunil

08/09/2008 12:25 am

I tried with no success. If Google is having some problem accessing my site, then it should prompt on my webmaster. In webmaster, sitemap is in pending state from many days and no Google employee wants to answer on this, They know there is a problem with many other blogs as well which are using custom domains. Such a big name and such a poor service, Next time, i am going to switch to wordpress and bye bye to Google.

lap nguyen

08/09/2008 07:29 am

My 3 blogspots can't be shown up for 2 weeks. I still can see it on another computer. My gmail acount is not working either> Please give me some direction. Thanks.

davina

08/10/2008 08:39 am

How to solve this problem?

Andy

08/12/2008 02:38 pm

I have a blog running on Wordpress. Since early August Google doesn't crawl it anymore. Rankings have dropped. Before then it was crawled daily and ranked well!

Doug Woodall

08/12/2008 11:23 pm

My Blog on Wordpress also has not shown in my Google alerts since early August. I did the latest Wordpress update about this time.

Sunil

08/14/2008 01:16 am

Hey Man, Finally got something good from Google. Sunil, I noticed that we just picked up a blog post from yesterday (I don't know which timezone you're in :-)) from your site, I think that's a good sign! Remember that it will generally take some time for the charts in Webmaster Tools to catch up as well but from what I can see it looks pretty much on-target. Great going. :) Everything back to Normal. Thanks Google :)

g@grandgood.com

08/20/2008 09:12 pm

kind of stinks. google stopped crawling our site but not the aggregrator blogs that use our site's content. so those come up in searches instead. weak.

RCHeli.dk

09/26/2008 09:21 pm

I have the exact same problem as g@grandgood.com, my latest post is not on google now, it was, but now it goon any solution for this issue ?

Boris

10/17/2010 06:22 am

it's still a problem

blog comments powered by Disqus