Yahoo! Slurp on the Loose?

Jan 17, 2007 - 7:57 am 5 by
Filed Under Yahoo SEO

A WebmasterWorld and Search Engine Watch Forums threads are both reporting issues with Yahoo! Slurp (Yahoo!'s Crawler) indexing pages they should not be, and in quantities that may be harmful.

It appears that only specific bots are not obeying the robots.txt file and indexing pages are rates that can potentially cause server issues.

The specific IP addresses appear to be in the 74.6.x block. They do reverse DNS to inktomi, which is correct.

Forum discussion at WebmasterWorld and Search Engine Watch Forums.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google March Core Update Done, HCU Recoveries, Site Reputation Abuse & AI Topics - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Ads

Google: Proximity Not A Relevancy Factor For Local Service Ads

May 10, 2024 - 7:21 am
Google

Google May Show How Many Shoppers Purchased On Your E-Commerce Site

May 10, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: May 9, 2024

May 9, 2024 - 4:00 pm
Google Updates

Google Search Ranking Update Volatility Starting On May 9th

May 9, 2024 - 7:51 am
Google

Google Renames AI Answer Back To AI Overview

May 9, 2024 - 7:41 am
Google

Google Tests New Search Notes Button

May 9, 2024 - 7:31 am
Previous Story: Wikipedia Search Engine Wikiseek