Craigslist Blocks Most Spiders: Millions of Pages Delisted

Jan 3, 2006 - 8:10 am 0 — by

A thread started at our forums named Craigslist Delists Millions of Pages from Search Engine Indexes uncovers the new robots.txt file in place over at Craigslist. It basically reads;

############################## # Exclude robots from these

User-agent: YahooFeedSeeker Disallow: /forums Disallow: /res/ Disallow: /post Disallow: /email.friend Disallow: /?flagCode Disallow: /ccc Disallow: /hhh Disallow: /sss Disallow: /bbb Disallow: /ggg Disallow: /jjj

User-agent: * Disallow: /cgi-bin Disallow: /cgi-secure Disallow: /forums Disallow: /search Disallow: /res/ Disallow: /post Disallow: /email.friend Disallow: /?flagCode Disallow: /ccc Disallow: /hhh Disallow: /sss Disallow: /bbb Disallow: /ggg Disallow: /jjj

#####################################

They supposedly had millions, 3.6 Million to be exact, of pages indexed at Google and millions at the other search engines. Now? 211,000 at Google, 280,000 at Yahoo and 4,695 atMSN.

Forum discussion at Search Engine Roundtable Forums.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Ads

Google Hotel Campaigns Migrating To Search Campaigns For Travel

Nov 18, 2026 - 7:11 am
Google

Google AI Overviews With Unfaded Citation Block

Nov 17, 2026 - 8:36 pm
Web Analytics

Google Tag & Google Tag Manager Unification

Nov 17, 2026 - 8:16 pm
Search Forum Recap

Daily Search Forum Recap: November 17, 2026

Nov 17, 2026 - 10:00 am
Search Engine Optimization

FBI: SEO Poisoning Led To $28 Million In Ransomware Payments

Nov 17, 2026 - 7:51 am
Google Search Engine Optimization

Google: A Site Move Doesn't Improve A Site's Overall Quality

Nov 17, 2026 - 7:41 am
Previous Story: Google's 2006 New Year Graphic; "Weird"?