Yahoo! Slurp Crawling Wild?

Jul 2, 2007 • 8:20 am | comments (3) by twitter Google+ | Filed Under Yahoo Search Engine Optimization

WebmasterWorld and DigitalPoint Forums thread report a spike in activity with Yahoo Search's web crawler, Slurp.

People have noticed a large increase in page hits and bandwidth usage, caused by Yahoo! Slurp recently.

I run several sites and I have the same problem. In fact, if Y! were to send me 1 visitor for every 10 bot visits I would need a dedicated server to handle the traffic :o)

i used to get like 4-5 at a time, but in the last few days it jumped like 8-10 and today it was an all time high of about 18 yahoo bots at a time. i havent checked the total of the bots/day but its surely going to be a larger number today

There are enough posts for me to report this, but not enough to say this is happening to a large percentage of site owners.

Forum discussion at WebmasterWorld and DigitalPoint Forums.

Previous story: Yahoo SmartAds Bridging Search Ads & Behavioral Targeting



07/02/2007 01:56 pm

post here ;) I'm almost tempted to block them and limit the number per minute..

Trapper Markelz

07/08/2007 10:43 pm

I run a site with a dedicated server, and I actually had to write special code to detect requests for search bots (like Yahoo Slurp) and prevent the page views... because it was slamming my database server and destroying my site performance. When I blocked the search spiders, it was a night and day difference. So while I am not getting the benefits of search engine traffic (and decreased ad revenue as a result) it is worth it because my site performance is SUPER fast without serving those spider visits.


03/18/2008 06:58 pm

Easier way. Use a robots.txt file. You can set a crawl delay and even target certain bots: User-agent: * Crawl-delay: 10 Disallow: User-agent: Slurp Crawl-delay: 300 yahoo is the worst limit them this way and you keep yourself in the search results and the bots do not crush your site. If there is still too many bots increase the delay. I would not go larger than 1440(one day though).

blog comments powered by Disqus