MSN Not Listening To Robots.txt Files? Or?

Jul 11, 2006 • 7:19 am | comments (0) by | Filed Under Bing Search

A Search Engine Roundtable Forums thread shows how MSN may not be properly listening to a robots.txt rule that Google set up for If you review Google's robots.txt file at you will see the standard, do not index command. But if you go to MSN Search and conduct a site command search for you will notice MSN has 14 results.

You will also notice those results are not full, they only contain the URL data. This is common, where the search engines will crawl just the URLs and not the content. I believe that may be the interpretation of the disallow. Take a look at Google, they are indexing URLs, and so is Yahoo! Search, and they are all doing it the same way.

Forum discussion at Search Engine Roundtable Forums.

Previous story: Google AdWords's Hidden Inactive Keywords
Ninja Banner
blog comments powered by Disqus