Entries from Search Engine Roundtable tagged with 'msnbot'

MSNBot Again Failing Reverse DNS Test

Last December, we reported that MSNBot was failing a reverse DNS lookup. Well, guess what folks - MSNBot is failing again on some IP addresses. An updated WebmasterWorld thread brought this to my attention and I verified it myself. Here...

MSRBot Bot, Microsoft Bot Causing Havoc on Some Servers?

There are two recent threads in different forums complaining about search engine spiders from Microsoft. A WebmasterWorld thread is specifically complaining about MSRbot, Microsoft's research bot, for attempting to crawl pages that don't exist. The other thread is from DigitalPoint...

Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler

The Live Search Blog announced several updated to their crawler. The first is a name change to reflect the upgrade, previously named msnbot/1.0, it is now named msnbot/1.1. The bulk of the changes include the HTTP Compression and Conditional Get...

Microsoft Confirms Reverse DNS Bot Issue & Promises To Resolve

Yesterday we reported that MSNBot failed some reverse DNS tests. In short, if you did a reverse DNS lookup on the IP addresses of their spiders, it would not resolve to search.live.com. The big issue with this is that Webmasters...

MSNBot Reverse DNS Test Fails Requirements

A year ago, Microsoft promised to enable Webmasters a method of verifying MSNbot. Way too often, rogue spiders mask themselves as official spiders from Google, Yahoo, Live Search or Ask.com. The search engines have enabled methods to conduct reverse DNS...

Live Search's Spam Fake Referrals Were Cloaking Tests, Says Microsoft

I have been corresponding with Microsoft about the weird spam-like referrals Live Search was sending to hundreds, if not thousands, of web site log files throughout the web. On September 6th, a Microsoft representative confirmed that these were actual tests...

MSNbot Adds Method For Verification

Microsoft has jumped on the bandwagon for allowing you to verify if the useragent, MSNBot, that is crawling your site, is truly from Microsoft or being spoofed by some content scraper. Google has released information in the past for verifying...

Live.com's Meta Tags For Crawl & Content Access Control

Sometimes it is hard to find the different supported Meta tags to help control where Live.com (MSNbot) can crawl on your site and what content they should cache. I dug up the resource on MSN's Site Owner Help section that...

MSN Crawlers Are Named

How cute, seriously, MSN has finally given names to their baby crawlers. You know, Google names their crawlers, i.e. GoogleBot, MediaBot, etc... Yahoo has Slurp, etc. Now MSN has named their crawlers. The MSN Shopping bot is msnbot-products. The MSN...


To subscribe to the Search Engine Roundtable, click here