Yahoo Blocking Googlebot From Yahoo Stores

Apr 14, 2011 • 8:08 am | comments (7) by twitter Google+ | Filed Under Google Search Engine Optimization
 

yahoo stopping googleAndrew Davis from the CPC Strategy blog informed me of a Google Merchant Center Help thread where dozens of Yahoo Store Merchants are complaining that their product images are not showing up in Google Image Search and the images are not showing up in Google Product Search and Product Ad results.

The reason appears to be because Yahoo recently decided to block Googlebot-Images from crawling the store's images. Do to this block, these merchants products won't show up in Google Image Search but more importantly is preventing the images and so other data from showing up in Google Product Search and the AdWords Product Ad results.

Andrew received a notification from Google that informed him of this, it read:

Hello,

Thank you for participating in Google Product Search. It has come to our attention that a robots.txt file is preventing us from crawling some or all of the images on your site. In order for us to access and display the images you provide in your product listings, we’d like you to modify your robots.txt file to allow user-agent ‘googlebot’ to crawl your site. Failure for Google to access your images may affect the visibility of your items on Google Product Search and Product Ad results.

To ensure the 'googlebot' is not being blocked, please add the following two lines of text to the end of your robots.txt file:

User-agent: googlebot

Disallow:

For more information on robots.txt files, please visit http://www.robotstxt.org. If you have any questions, please contact your webmaster directly.

Sincerely,

The Google Product Search Team

One person received a response from Yahoo on this where Yahoo downplayed the importance of merchants needing Google and other search engines from accessing and indexing their images. Yahoo said:

There is not any really good reason to have all of your images on Google image search. The only big reason for that is to grab images that are free and not link to your site, I can see it helping every so often but not worth the time or money at this time to go after. But that is just me. I have not seen any thing that supports merchants in Google image search.

Seriously? Yahoo! Seriously?

Forum discussion at Google Merchant Center Help.

Previous story: Daily Search Forum Recap: April 13, 2011
 

Comments:

Robert Lloyd

04/14/2011 12:13 pm

I don't see how this is helping anyone.

jérémy allard

04/14/2011 12:18 pm

WTF ? it's funny :D

scottclark

04/14/2011 05:52 pm

I've inquired to Yahoo-Inc about this. I do SEO on Yahoo! Stores in their developer partner program - so will try to get to the bottom of it if possible.

John Smith

04/14/2011 06:22 pm

This seems to be a non-issue. Found this: http://dashboard.feedwizards.com/news/google/productimagewarning "..the net result is that Google can access the images, and that this warning can be safely ignored. We will continue to monitor and if there is anything we need to adjust, we will. If you see any issues, let us know, but as of this time we see no reason for concern."

David Weichel

04/14/2011 09:57 pm

Thanks for sharing our blog post Barry. The issue here is that Yahoo is listing a robots.txt file in a subdirectory of their image server for whatever reason. Any robots.txt file in a subdirectory is automatically rendered invalid, which prompts the question: why does Yahoo have it there to begin with? The email from Google is could be the result of either two things: 1.) There’s an apache header or 301 redirect that’s pointing Yahoo’s robots.txt file (ep.yimg.com/ca/robots.txt) to their main directory which caused the googlebot to read it and ignore those files. This would be a huge mistake on Yahoo Stores, obviously. 2.) Googlebot experienced a bug and acknowledged robots.txt files that should never have been read (most likely the case). Either way, we will be keep closing tabs on this robots.txt issue and will post any relevant updates on our blog.

Manny

04/14/2011 10:04 pm

That's why Yahoo is not the SE #1...

Chavarcal

04/15/2011 10:49 am

Its a google change http://dashboard.feedwizards.com/news/google/productimagewarning

blog comments powered by Disqus