(2)
There is a WebmasterWorld thread where folks are reporting that MSN Search is indexing a bit too much. Reports suggest that MSN is crawling and indexing the content found on pages which are excluded via the robots.txt directive.
Normally, if a search engine crawls the page, they may only keep the URL in their index but not the content.
Moderator Receptional adds, "My examples show no snippet. But I can get them by typing unique text into the search query."
So if they are returning results for text found on the page, maybe they are indexing a bit too much?
On the other hand, I have a thread at Search Engine Roundtable Forums suggesting MSN doesn't index enough, specifically new sites...
Forum discussion at WebmasterWorld & Search Engine Roundtable Forums.
Comments:
Marcia
01/08/2007 08:03 pm
This has happened with other search engines, the URL is there because it was found as a link on another page and put into the crawl queue.
George
01/13/2007 02:10 am
I have HUGE peoblems with MSN and their 301 redirects. Pages have been removed and cached pages just say that they have been Permanently Moved which is not true. I used TranslationGold to translate my pages and that seems have caused me a lot of problems. Pages with query strings are not longer indexed and show up as 301'ned.... Not sure what is going on but it has happened to all my sites :(