Some Webmasters Are Banning Yahoo Slurp

Jun 30, 2008 • 7:35 am | comments (16) by twitter Google+ | Filed Under Yahoo Search Engine Optimization

It seems like some webmasters are becoming fed up with the activity of Yahoo's crawl, Yahoo Slurp, relative to the amount of traffic Yahoo Search is sending the web site. In fact, some webmasters have taken the plunge and banned Yahoo from crawling their sites.

It is funny, because some webmasters were or are afraid to ban Yahoo because they think it might have some sort of impact on their Google results. One such webmaster got over it and decided to ban Yahoo despite his fears.

There is a thread at WebmasterWorld discussing the pros and cons. Some also wonder that will banning Yahoo have a negative impact on the Yahoo SearchScan & McCafee partnership and might have your site marked as harmful? I doubt it.

If you want to block Yahoo Slurp, learn how to at this page.

Forum discussion at WebmasterWorld.

Previous story: Moving to a U.S Hosted Server: Will it Impact Country Specific SEO?


No Name

06/30/2008 12:34 pm

Where I get the lions share of my traffic from Google, the conversion rates I get from Yahoo are so much better that I get from anywhere else. I will keep Slurp coming around for a while.


06/30/2008 02:30 pm

I agree with the previous commentor. Although the traffic Yahoo! sends is small in volume (about 8% or so for most of my sites) the conversion rate and the average order value this traffic delivers is generally better. I'm quite happy with others pulling out of Yahoo! makes my positions better :p

No Name

06/30/2008 03:06 pm

Those who block are ..

Michael Martinez

06/30/2008 04:06 pm

I have banned Slurp from my forums many times through the years. It tends to get caught up in the crawling process. But after a couple of weeks I'll let it back in. Once or twice I have had to ban Googlebot, too. I think this is a problem endemic to large complex sites with a lot of tight interlinkage, such as you have on forums and blogs.

Rob Abdul

07/01/2008 08:57 am

It could be that Yahoo is trying something new with Slurp and the frequency of visits has gone up for that reason.

Rob Abdul

07/01/2008 08:58 am

It could be that Yahoo is trying something new with Slurp and the frequency of visits has gone up for that reason.

Jeff Brown

07/02/2008 06:12 am

As for me I recieve better returns from Yahoo I believe for my business it is Demographics.

Ruben Zevallos Jr.

07/02/2008 10:59 am

It's very interesting, because Live and Yahoo did not bring much users to my web sites... but they are there almost the same weight than Google... so... I'll take the idea of User-agent: Slurp Crawl-delay: 240 And use for both and see...


07/02/2008 02:38 pm

I don't see the problem. Traffic is traffic, it is a numbers game. The more traffic you get, the more sales you will make in the long run.


07/03/2008 04:21 pm

I really do hope Yahoo makes a come back. One search engine dominating us all is not a good thing for the market.

No Name

07/03/2008 07:46 pm

I agree with previous poster. Although I get about 4x more traffic from Google, my conversion numbers are around the same. Yahoo does have better targeted traffic.


07/03/2008 08:19 pm

We get 90% of our traffic from Google and only 5% from Yahoo...We will keep them around for now.


07/19/2008 12:33 am

Apparently I have had hundreds of GB of traffic going to yahoo slurp on my site I needed this to stop IMMEDIATELY.... I believe it is because slurp for some reason is downloading my FLV files - I have no clue why it would need to do this but whatever..... So I am testing some new code in my robots.txt file that hopefully should eliminate this problem.... user-agent: * Disallow: /flvplayer.swf User-agent: Googlebot Disallow: /*.flv$ User-Agent: Yahoo! Slurp Disallow: /*.flv$ So if you have a website with various media files, you may be able to tell the bot to not download them. Here are the resources I used to build these directives.

No Name

07/25/2008 05:15 pm

I've often heard different Webmasters rant an rave about the virtues of different technologies. One likes article marketing while another can't live without PPC. In my opinion you need all of the above to succeed in Internet Marketing. Just because one technology may out perform another that is in and of itself no reason to ban that technology. Perhaps give more time to one activity as compared to another; but altogether banning has never made sense to me.

No Name

08/17/2008 12:12 pm

Lol, seems to me that if Yahoo traffic converts so well y'all might be better off optimizing for Yahoo and banning all those visitors from google, who don't make you any money. It's not a "numbers game" it's a Highly targetd, converting traffic, game" and whoever sends the best converting traffic is the area I'm going to focus on. Another thing you y'all might want to consider, regarding your google listings, is tightening them up so the traffic is better targeted. Don't blame the SE for your traffic problems nor the traffic they send. The SE didn't optimize your pages, you did.

No Name

11/10/2008 08:41 pm

Does yahoo even listen to robot.txt??

blog comments powered by Disqus