Yahoo SmartAds Bridging Search Ads & Behavioral Targeting | Main | Google Accounts Security Issue Confuses User Authentication

July 2, 2007

Yahoo! Slurp Crawling Wild?

WebmasterWorld and DigitalPoint Forums thread report a spike in activity with Yahoo Search's web crawler, Slurp.

People have noticed a large increase in page hits and bandwidth usage, caused by Yahoo! Slurp recently.

I run several sites and I have the same problem. In fact, if Y! were to send me 1 visitor for every 10 bot visits I would need a dedicated server to handle the traffic :o)
i used to get like 4-5 at a time, but in the last few days it jumped like 8-10 and today it was an all time high of about 18 yahoo bots at a time. i havent checked the total of the bots/day but its surely going to be a larger number today

There are enough posts for me to report this, but not enough to say this is happening to a large percentage of site owners.

Forum discussion at WebmasterWorld and DigitalPoint Forums.



Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!

posted rustybrick in Yahoo! Search Optimization at July 2, 2007 8:20 AM Comments (3)

• Google Co-op Cartoon Barry

Comments

post here ;)
http://forums.seroundtable.com/showthread.php?t=2011

I'm almost tempted to block them and limit the number per minute..

 

Posted by gabs at July 2, 2007 09:56

I run a site with a dedicated server, and I actually had to write special code to detect requests for search bots (like Yahoo Slurp) and prevent the page views... because it was slamming my database server and destroying my site performance.

When I blocked the search spiders, it was a night and day difference. So while I am not getting the benefits of search engine traffic (and decreased ad revenue as a result) it is worth it because my site performance is SUPER fast without serving those spider visits.

 

Posted by Trapper Markelz at July 8, 2007 18:43

Easier way. Use a robots.txt file. You can set a crawl delay and even target certain bots:

User-agent: *
Crawl-delay: 10
Disallow:
User-agent: Slurp
Crawl-delay: 300

yahoo is the worst behaved..you limit them this way and you keep yourself in the search results and the bots do not crush your site. If there is still too many bots increase the delay. I would not go larger than 1440(one day though).

 

Posted by William at March 18, 2008 14:58

Post a comment (Note: Can Take 120 Seconds For Your Comment To Show Up)

Do you want us to save your personal Information?

Premium Sponsors + advertise

To subscribe to the Search Engine Roundtable, click here