Craigslist Blocks Most Spiders: Millions of Pages Delisted

Jan 3, 2006 - 8:10 am 0 by

A thread started at our forums named Craigslist Delists Millions of Pages from Search Engine Indexes uncovers the new robots.txt file in place over at Craigslist. It basically reads;

############################## # Exclude robots from these

User-agent: YahooFeedSeeker Disallow: /forums Disallow: /res/ Disallow: /post Disallow: /email.friend Disallow: /?flagCode Disallow: /ccc Disallow: /hhh Disallow: /sss Disallow: /bbb Disallow: /ggg Disallow: /jjj

User-agent: * Disallow: /cgi-bin Disallow: /cgi-secure Disallow: /forums Disallow: /search Disallow: /res/ Disallow: /post Disallow: /email.friend Disallow: /?flagCode Disallow: /ccc Disallow: /hhh Disallow: /sss Disallow: /bbb Disallow: /ggg Disallow: /jjj

#####################################

They supposedly had millions, 3.6 Million to be exact, of pages indexed at Google and millions at the other search engines. Now? 211,000 at Google, 280,000 at Yahoo and 4,695 atMSN.

Forum discussion at Search Engine Roundtable Forums.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 25, 2024

Jul 25, 2024 - 10:00 am
Google Ads

Google Again: We Will Test Ads In AI Overviews Soon

Jul 25, 2024 - 7:51 am
Bing Search

Microsoft Now Testing Bing Generative Search Experience

Jul 25, 2024 - 7:41 am
Bing SEO

Reddit Blocked Bing Search & Others But Not Google

Jul 25, 2024 - 7:31 am
Local Search

Apple Maps Web Version Launches Beta

Jul 25, 2024 - 7:21 am
Google Ads

Google Local Service Ads Shows Phone Number On Hover

Jul 25, 2024 - 7:11 am
Previous Story: Google's 2006 New Year Graphic; "Weird"?