MSN Indexing Pages With No Index Commands?
There is a WebmasterWorld thread where folks are reporting that MSN Search is indexing a bit too much. Reports suggest that MSN is crawling and indexing the content found on pages which are excluded via the robots.txt directive.
Normally, if a search engine crawls the page, they may only keep the URL in their index but not the content.
Moderator Receptional adds, "My examples show no snippet. But I can get them by typing unique text into the search query."
So if they are returning results for text found on the page, maybe they are indexing a bit too much?
On the other hand, I have a thread at Search Engine Roundtable Forums suggesting MSN doesn't index enough, specifically new sites...
Forum discussion at WebmasterWorld & Search Engine Roundtable Forums.
Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!
rustybrick in Microsoft MSN Search at January 5, 2007 7:07 AM
Comments (2)

Comments
This has happened with other search engines, the URL is there because it was found as a link on another page and put into the crawl queue.
Posted by Marcia at January 8, 2007 15:03