Yahoo Slurp, Yahoo Search Crawler, Suffering From ADHD?

Mar 31, 2008 • 7:48 am | comments (3) by twitter Google+ | Filed Under Yahoo Search Engine Optimization
 

A DigitalPoint Forums thread has dozens of reports that Yahoo Search's crawler, Yahoo Slurp, took some bad medicine recently. Many are reporting that they see the crawler spidering their sites like never before. Some times they have seen the spider multiple to over a 1,000 crawls at one time.

The first report claimed about 500 Yahoo spiders:

I have a forum and I get 500+ (sometimes 800+) Yahoo spiders daily. Why is there so many? I only get 1 or 2 Google spiders.

Others reported similar cases with much more Yahoo spider activity then Google spider activity. These reports seemed to have died down recently.

The initial report came at Friday afternoon and then died out Saturday afternoon. So maybe it was just a temporary Yahoo glitch?

Forum discussion at DigitalPoint Forums.

Update: Yahoo sent me a statement update this behavior:

On March 29th and 30th, some of you have noticed Yahoo! Slurp spidering your website more than usual – this temporary blip was inadvertently caused by a major crawl infrastructure upgrade we have been doing for last 1 month.

While the Yahoo! Slurp should be stable now, if you continue to notice unreasonable traffic from Yahoo! spiders, please provide feedback at the Site Explorer Suggestion Board.

Thanks,

Yahoo! Search Team

Previous story: Microsoft ContentAds Publisher Program Pilot Sign Up Form
 

Comments:

minstrel

03/31/2008 02:32 pm

This is nothing new. This has been true for as long as I can remember.

Barry Schwartz

03/31/2008 03:03 pm

Not at this level...

No Name

03/10/2009 08:05 am

Some advanced browsers can use yahoo or google crawler trace sign. Some download manager use g or y trace to explore your website and make illegal download of content. Or yahoo just testing new algorithms & equipment

blog comments powered by Disqus