MSN Indexing Pages With No Index Commands?

Jan 5, 2007 • 7:07 am | comments (2) by twitter Google+ | Filed Under Bing Search
 

There is a WebmasterWorld thread where folks are reporting that MSN Search is indexing a bit too much. Reports suggest that MSN is crawling and indexing the content found on pages which are excluded via the robots.txt directive.

Normally, if a search engine crawls the page, they may only keep the URL in their index but not the content.

Moderator Receptional adds, "My examples show no snippet. But I can get them by typing unique text into the search query."

So if they are returning results for text found on the page, maybe they are indexing a bit too much?

On the other hand, I have a thread at Search Engine Roundtable Forums suggesting MSN doesn't index enough, specifically new sites...

Forum discussion at WebmasterWorld & Search Engine Roundtable Forums.

Previous story: Idea To Place Google Image Ads Near Google Text Ads As Alternate
 

Comments:

Marcia

01/08/2007 08:03 pm

This has happened with other search engines, the URL is there because it was found as a link on another page and put into the crawl queue.

George

01/13/2007 02:10 am

I have HUGE peoblems with MSN and their 301 redirects. Pages have been removed and cached pages just say that they have been Permanently Moved which is not true. I used TranslationGold to translate my pages and that seems have caused me a lot of problems. Pages with query strings are not longer indexed and show up as 301'ned.... Not sure what is going on but it has happened to all my sites :(

blog comments powered by Disqus