MSN Not Listening To Robots.txt Files? Or?

Jul 11, 2006 - 7:19 am 0 by
Filed Under Bing Search

A Search Engine Roundtable Forums thread shows how MSN may not be properly listening to a robots.txt rule that Google set up for api.google.com. If you review Google's robots.txt file at http://api.google.com/robots.txt you will see the standard, do not index command. But if you go to MSN Search and conduct a site command search for site:api.google.com you will notice MSN has 14 results.

You will also notice those results are not full, they only contain the URL data. This is common, where the search engines will crawl just the URLs and not the content. I believe that may be the interpretation of the disallow. Take a look at Google, they are indexing site:api.google.com URLs, and so is Yahoo! Search, and they are all doing it the same way.

Forum discussion at Search Engine Roundtable Forums.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Video Recaps

Search News Buzz Video Recap: Google June Spam Update Done, Fraud DMCA Takedowns Breaks Google, Costly Google Ad Budget Changes & More

Jul 3, 2026 - 8:01 am
Google Maps

Google Local Reviews Go Missing For Many Businesses

Jul 3, 2026 - 7:51 am
Google Maps

Google Business Profile Restrictions & Penalties Are Additive

Jul 3, 2026 - 7:41 am
Google Maps

Google Tests New Local Places Layout Design

Jul 3, 2026 - 7:31 am
Google

Google Fourth Of July 2026 Doodle Celebrating 250 Years Of The USA

Jul 3, 2026 - 7:23 am
Google Ads

Google Third-Party Rates For Hotel Ads Feature Going Away On Sept. 30

Jul 3, 2026 - 7:21 am
 
Previous Story: Google AdWords's Hidden Inactive Keywords