A DigitalPoint Forums thread has dozens of reports that Yahoo Search's crawler, Yahoo Slurp, took some bad medicine recently. Many are reporting that they see the crawler spidering their sites like never before. Some times they have seen the spider multiple to over a 1,000 crawls at one time.
The first report claimed about 500 Yahoo spiders:
I have a forum and I get 500+ (sometimes 800+) Yahoo spiders daily. Why is there so many? I only get 1 or 2 Google spiders.
Others reported similar cases with much more Yahoo spider activity then Google spider activity. These reports seemed to have died down recently.
The initial report came at Friday afternoon and then died out Saturday afternoon. So maybe it was just a temporary Yahoo glitch?
Forum discussion at DigitalPoint Forums.
Update: Yahoo sent me a statement update this behavior:
On March 29th and 30th, some of you have noticed Yahoo! Slurp spidering your website more than usual – this temporary blip was inadvertently caused by a major crawl infrastructure upgrade we have been doing for last 1 month.While the Yahoo! Slurp should be stable now, if you continue to notice unreasonable traffic from Yahoo! spiders, please provide feedback at the Site Explorer Suggestion Board.
Thanks,
Yahoo! Search Team
