(13)
A new update has been made to the Google webmaster guidelines document, which describes you should not block the Google ads destination URL with your robots.txt file.
The new text is under the technical guidelines and reads:
Make efforts to ensure that a robots.txt file does not block a destination URL for a Google Ad product. Adding such a block can disable or disadvantage the Ad.
The guideline before somewhat implies the complete opposite:
Make reasonable efforts to ensure that advertisements do not affect search engine rankings. For example, Google's AdSense ads and DoubleClick links are blocked from being crawled by a robots.txt file.
So what do you make of this new change to the Google webmaster guidelines?
This was spotted first by Menashe Avramov.
Forum discussion at Google+.
Update: The following day, Google has removed the language from the guidelines. I guess they decided it was confusing also?
Comments:
mvandemar
02/13/2014 02:42 pm
Those are 2 different things. A "a destination URL for a Google Ad product" is a landing page for something that is being advertised on Google AdWords, which they want to be able to spider to determine the Quality Score, whereas the second paragraph is saying make sure that outgoing links do not pass PageRank. AdSense links are blocked, eg. on google.com/robots.txt: Disallow: /aclk and on ad.doubleclick.net/robots.txt: User-agent: * Disallow: / I do find it a little odd that www.googleadservices.com/robots.txt doesn't exist. Either way though, those guidelines are not in conflict with one another.
StevenLockey
02/13/2014 04:46 pm
Yep, made sense to me, but I had to read it a few times. Perhaps they should re-word it to clarify both a little better.
Barry Schwartz
02/13/2014 04:48 pm
yea, that is what I thought but worded so weird.
Michael Martinez
02/13/2014 05:12 pm
This is stupid. They need to use separate crawling agents so that people don't have to NoIndex their Adwords landing pages.
Barry Schwartz
02/13/2014 05:13 pm
Yea, I thought GoogleBot Media or something was for that.
Paul Ryan
02/13/2014 10:07 pm
They do, but some people just do not think about it an indiscriminately block. Unfortunately, common sense does not prevail. This is not new advice from Google. See: http://adsense.blogspot.co.uk/2011/11/adsense-crawler-errors-check-your.html https://support.google.com/adsense/answer/10532?hl=en-GB What Barry posted is the webmaster guidelines. It's not new advice for AdSense. They probably added it in there, too, because too many people were ignoring it and then crying.
Barry Schwartz
02/13/2014 10:08 pm
But they should specify the media bot, because it is somewhat confusing.
Jordan Silton
02/13/2014 11:14 pm
Seems like the timing could be related to the page layout algorithm change. Would Google really respect robots.txt in a case where there were a ton of ad units at the top of the page? It's possible.
Michael Martinez
02/14/2014 02:30 am
It doesn't matter what they specify. All the Googlebot user-agents contribute/draw from a pool of recently crawled URLs -- or so you and others have reported Matt Cutts as saying in years past. So it doesn't matter which of the existing user-agents they use, they're all mostly blocked if you're only blocking GoogleBot. It would be better to handle the advertising landing pages completely separately from the rest of the Google services. Now, if they want to publish a corrective statement that clears this up for everyone, I'll be glad to see that.
Paul Ryan
02/14/2014 09:12 am
Why? They link to: https://support.google.com/webmasters/answer/156449 Which explains it all, including specifying Mediapartners-Google. If it's AdWords then that is a different matter (as the bot is Adsbot-Google). That will ignore robots.txt unless Adsbot-Google is specifically specified: https://support.google.com/adwords/answer/2404197?hl=en-GB
Barry Schwartz
02/14/2014 06:02 pm
FYI, the guidelines were updated this morning again and Google removed that new guideline. Updated story at http://www.seroundtable.com/google-webmaster-guidelines-updated-18117.html
Lathesis
02/18/2014 07:15 am
why shouldn't block Google ads destination URL with your robots.txt file ? What is the purpose behind that ? what google want to do with a web admin rights ? All question comes up in mind in a Que !!
HymnSite
06/21/2014 03:19 pm
I would appreciate some clear direction on this. If we do what Google wants us to do, what should appear in my robots.txt file with respect to AdSense and DoubleClick?