Google's New Search Snippets Blocked By Robots.txt Warning

Aug 16, 2012 • 8:23 am | comments (34) by twitter Google+ | Filed Under Google Search Engine Optimization
 

A description for this result is not available because of this site's robots.txt

Google is showing a more descriptive search snippet for when it cannot show any search snippet in the search results.

If a page is being blocked by the robots.txt file, Google won't show the search snippet (description) in the search results. Google may show the page in the search results because of the linkage data it has. The example in the picture above is the only URL I have blocked from Google indexing it. I regret writing it, but also would regret removing it, so I blocked it from Google.

If you try to match on it within Google, it turns the notification:

A description for this result is not available because of this site's robots.txt – learn more.

The learn more link goes to the page named Block or remove pages using a robots.txt file in Google's help section.

I think this provides a much better search experience for someone who sees these blocked pages.

I do wonder if they show up for non site: command searches, I'd assume so.

Forum discussion at WebmasterWorld.

Previous story: Email Delivery Bug With Google Webmaster Tools Messages
 

Comments:

Martin

08/16/2012 01:05 pm

Hello Barry, I'm glad to read your post because I have this problem and I didn't find any anwser. I'm using Wordpress and my homepage is not blocked in Robots.txt but Google display "A description for this result is not available because of this site's robots.txt" in serps. Have any idea? Maybe I should wait a little?

Robin331

08/16/2012 01:12 pm

this is bullshit. If page was blocked at robots.txt - them not able to display it search results for any reasons. It can be copyright reasons, personal choice, etc. But google not have any rights to display this page in search results. Or monster stop worry about anything related copyrights???

Webstats Art

08/16/2012 01:36 pm

Googlebot is crazy by the fact that it needs to feed all the time. If you make a site full of coherent rubbish, it will be spidered a lot (like crazy) as long as you can keep uploading the nonsense. You will get traffic too... from those folks who like to see what was uploaded in the last 24 hours. Haha! There is a serious need for google to fix some things but alas they are far from it right now.

Steve Peron

08/16/2012 03:50 pm

I dont think Google has this new feature working correctly. I am seeing this message for a page that is 404ed and it is not blocked in the robots.txt.

John Britsios

08/17/2012 05:03 am

Why do you block the page in the robots.txt? I would rather go for . Wouldn't that probably work? Or am I suffering again from a "Buffer Overflow"?

John Britsios

08/17/2012 05:06 am

I fully disagree! Snippets may show up in the search results for pages blocked via robots.txt which have links pointing to them. I know that since years and seen already frequently. Also Matt Cutts explained that on one of his videos if you would like to disagree with me and you trust Matt.

Guest

08/17/2012 05:07 am

Some discussion in Barry Schwartz’ post on Search Engine Roundtable about Matt Cutts’ comments at SESSF, as well as comments and clarifications added later, make it sound as if we can expect some help from Google later this year in determining exactly what needs to be done to recover from penalties. This will be a VERY welcome improvement, to help avoid situations such as Doc Sheldon detailed in his post, Stop Playing Games, Google!

bob

08/17/2012 06:19 am

I just disagree with idea, and looks like with M. Cutts and you. If page was blocked from google bot, it mean what content owner not want it indexed or displayed in search results. Technically google not have any right (links to this page not give any right to google) to display this content.

John Britsios

08/17/2012 07:44 am

Bob we are talking about snippets. Do a test and you will see who is right.

Colin

08/17/2012 11:08 am

John is correct. If you want a page not to appear use a noindex tag, not robots.txt.

winemp3

09/01/2012 02:03 am

i've even allow the sitemap.xml for all user-agents still google shows description not available here's the robots.txt file www.winemp3.com/robots.txt

Sanjay

09/05/2012 04:53 am

How will this effect work for blogspot. for a newly created blogspot in indexed pages, i can see the label indexed pages with saying that description is blocked due to robot.txt file.How can we place a description for labels. although we cant place descrption how to replace it with a description

kind-broker.ru

09/10/2012 06:49 am

I have no robots.txt restrictions but google shows this message for my site..

Mike Hale

09/21/2012 04:04 pm

Hey Guys... have you checked what is in the robots.txt ... default WP robots.txt shows to disallow from wp-admin so if you are using a plugin to display your SEO such as your description and such and the plugin is trying to deliver the meta tags via the wp-admin then the crawlers will not find it. I would recommend trying an alternative method for delivering your meta tags (like a different plugin or otherwise) to ensure that is not the culprit. Also might want to check .htaccess or your theme files to make sure that the theme is not interfering with the bot trying to read that particular meta tag. If you can see that robots.txt is clean and can even see how the bot reads it and it is clean then it has to be something else.

Raj

10/12/2012 05:29 am

I agree, Google have no any rights to display the private pages in search results.

Trixit

10/12/2012 05:09 pm

Google should not even be putting that msg up in the first place. it does not matter if the developer wanted to hide description or not. If there isn't one, there should be nothing shown. that msg is just unnecessary. You people try defend google, for something that none of their business. id a developer wants to know they will learn.

John Brown

10/12/2012 05:56 pm

it not important snippet or not. Website owner not wish his page in search results. And him placed disallow tag to tell google about it. But in light of last google updates, I think need to prohibit google from indexing websites by default (legal way). And allow google to index websites only if website owner agrees with it and added website to gwt for example.

Vikas

10/31/2012 01:27 am

Hi Martin, You should have a look at your site's privacy settings if you think there's nothing wrong with robots.txt

Vikas

10/31/2012 01:30 am

Man, did you notice your robots.txt got indexed? Congratulations!!

John Britsios

10/31/2012 07:51 am

Vikas and so what is the problem with that? Did you check the robots.txt of the White House? It is indexed too, and has PageRank 4. To make sure that different files types do not get indexed, if you are on Apache you can add the following lines into your .htaccess file and your problem is solved: Header set X-Robots-Tag "noindex,noarchive,nosnippet,follow"

Daniel Ivanitski

11/10/2012 07:35 am

I have done everything.... Robots.txt ----------------------------- User-agent: Googlebot Disallow: /*.swf$ --------------------------------- and .htaccess ----------------------------------- Header set X-Robots-Tag "noindex,nofollow" ------------------------------------ and I still get all my Flash (SWF) files indexed with that message. So what is next... what is the answer.... ???

Philip

11/15/2012 09:00 pm

What I'm trying to understand is why one of my product review posts show the description message... "A description for this result is not available because of this site's robots.txt." I didn't block this post or any of my posts in my robots.txt file. Also, my product review post use to appear on page one on Google for many of my keywords but seems to have gone to Google's supplemental index now. I had no problem with any of my descriptions appearing in the SERPs before. I have been using the All-in-One WP plugin for years so I don't see how there's a problem with it since all my other post descriptions appear normal in the SERPs. The only change I made as of late was a 307 temporary redirect on that product review post when doing some redesigning work. A couple days ago, I removed the temporary redirect. and still my product review post is experiencing these issues. I can only assume this has something to do with the 307 temporary redirect and I have to wait on Google to set everything right -- unless someone here thinks something else is wrong.

Philip

11/17/2012 12:07 pm

Looks like everything is back to norm now. I'm no longer have the missing description issue and my product review post is back on page one of Google for my chosen keywords. So me issue was related to the 307 temporary redirect that I removed a few days ago..

Brothers Three

01/19/2013 07:39 am

I have this message showing up for a WordPress page which has no robots.txt file on the server. I can't imaging that the WordPress .htaccess file would cause this message to appear, would it?

Brothers Three

01/19/2013 07:42 am

I have a WordPress page which returns this "robots.txt" message in a Google search, yet there is no robots.txt file on the server. Doesn't make sense at all.

MTS

03/23/2013 08:39 pm

this is terrible. my website has been blocked because of this dumb robots.txt thing. It can't display a description or it wont show up when i search it either and i have NO IDEA how to remove this idiotic robots.txt.

Yogita Aggarwal

06/12/2013 06:28 am

My blog is facing this issue... Don't know what to do now? Anyone please help. Google is showing number of omitted results for my blog. I have corrected my robots file as well still no response.

Drew Leahy

08/02/2013 10:42 pm

WOW! Thank you! This is exactly what was wrong :)

Cyndy Cox

09/08/2013 12:53 am

I am receiving this message when I try to view google maps on my mac.

nomad411

10/23/2013 11:43 pm

No robots.txt is the same as having one saying Block everything. You need to put one in with something like this: User-Agent: * Disallow: /wp-content/plugins/

Michael Buluma

12/05/2013 08:37 am

hi. how did you solve that?

Mohith Agadi

01/31/2014 06:45 am

If there is no robots.txt, that means we're not saying any bot anything. By default any bot can fetch/access/index the pages. Robots.txt is required only when we have to block bots from crawling/indexing specific directories or pages.

Gerald

02/13/2014 09:04 pm

I hate those hippies over in Mountianview

ajaniashish

07/23/2014 06:30 am

Having the same problem in one of my website built using WordPress. There is also a robots.txt placed in root but with: User-agent: * Disallow: / Modified it with code like: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ And it works by showing correct description in google search results. Hope this comment can help someone else too.

blog comments powered by Disqus