Entries from Search Engine Roundtable tagged with 'msnbot'

We have more MSNBot troubles to unfortunately bring to you. Microsoft Bing's spider, MSNBot, is apparently not listening to directives they should be listening to. In this case, it is the crawl delay command, where a couple users are claiming...

Posted on November 11, 2009 8:49 AM

Most search spiders have been known to get a bit crawl happy from time to time. But the most complaints over time come from MSNBot which tends to often get out of hand and send their spiders on individuals sites...

Posted on October 23, 2009 8:20 AM

With all the on going issues with MSNBot not behaving, I am not too surprised to see more complaints about the little spider. New confirmed reports from Bing Forums shows that MSNBot is hiding itself under the UserAgent of Mozilla/4.0....

Posted on September 17, 2009 8:31 AM

Over the past few weeks, I have been noticing threads pop up in the Bing forums with complaints from webmasters that Bing's bot, aka MSNBot is not honoring their robots.txt directives. It was not just one thread, but at least...

Posted on September 9, 2009 8:45 AM

Yesterday, I had a quick chat with a Microsoft Bing webmaster representative, where I asked him several questions that have been recently buzzing around the webmaster community. One of those questions was, is MSNbot clicking on Bing adCenter ads? The...

Posted on August 13, 2009 8:14 AM

Shawn Hogan, DigitalPoint's founder, has posted a thread at DigitalPoint Forums clearly showing his frustration with MSNBot, Microsoft Bing's search crawler. He is upset that the bot is crawling too much, too fast - causing an unnecessary spike in load...

Posted on August 3, 2009 8:19 AM

There is a thread I have been watching at the Bing Community where one member said that he had log files that shows MSNBot (Microsoft Bing's crawler) is clicking on Microsoft adCenter search ads, possibly charging him for those clicks....

Posted on July 29, 2009 9:21 AM

A Bing thread has two complaints from webmasters on Microsoft's newish crawler, MSNBot 2. The bot, in some cases, might be adding a pound sign (#) to the end of the URLs. Now, this might not be the fault of...

Posted on July 20, 2009 8:54 AM

There are several reports around the web about a new search bot by Microsoft that is causing major issues for web servers. The bot is named adidxbot and the useragent looks like this: adidxbot/1.1 (+http://search.msn.com/msnbot.htm). This bot has been on...

Posted on July 7, 2009 8:08 AM

incrediBILL, moderator at WebmasterWorld, noticed that one of Live Search's bots was crawling through his JavaScript. The bot is named MSNBOT-MEDIA and he noticed that it was accessing JavaScript files and AJAX functions. He noticed that the bot was triggering...

Posted on January 2, 2009 7:49 AM

Last December, we reported that MSNBot was failing a reverse DNS lookup. Well, guess what folks - MSNBot is failing again on some IP addresses. An updated WebmasterWorld thread brought this to my attention and I verified it myself. Here...

Posted on March 11, 2008 6:59 AM

There are two recent threads in different forums complaining about search engine spiders from Microsoft. A WebmasterWorld thread is specifically complaining about MSRbot, Microsoft's research bot, for attempting to crawl pages that don't exist. The other thread is from DigitalPoint...

Posted on February 25, 2008 7:18 AM

The Live Search Blog announced several updated to their crawler. The first is a name change to reflect the upgrade, previously named msnbot/1.0, it is now named msnbot/1.1. The bulk of the changes include the HTTP Compression and Conditional Get...

Posted on February 13, 2008 7:28 AM

Yesterday we reported that MSNBot failed some reverse DNS tests. In short, if you did a reverse DNS lookup on the IP addresses of their spiders, it would not resolve to search.live.com. The big issue with this is that Webmasters...

Posted on December 6, 2007 7:20 AM

A year ago, Microsoft promised to enable Webmasters a method of verifying MSNbot. Way too often, rogue spiders mask themselves as official spiders from Google, Yahoo, Live Search or Ask.com. The search engines have enabled methods to conduct reverse DNS...

Posted on December 5, 2007 7:22 AM

I have been corresponding with Microsoft about the weird spam-like referrals Live Search was sending to hundreds, if not thousands, of web site log files throughout the web. On September 6th, a Microsoft representative confirmed that these were actual tests...

Posted on December 4, 2007 3:00 PM

Microsoft has jumped on the bandwagon for allowing you to verify if the useragent, MSNBot, that is crawling your site, is truly from Microsoft or being spoofed by some content scraper. Google has released information in the past for verifying...

Posted on November 30, 2006 8:17 AM

Sometimes it is hard to find the different supported Meta tags to help control where Live.com (MSNbot) can crawl on your site and what content they should cache. I dug up the resource on MSN's Site Owner Help section that...

Posted on October 6, 2006 7:54 AM

How cute, seriously, MSN has finally given names to their baby crawlers. You know, Google names their crawlers, i.e. GoogleBot, MediaBot, etc... Yahoo has Slurp, etc. Now MSN has named their crawlers. The MSN Shopping bot is msnbot-products. The MSN...

Posted on August 1, 2006 7:36 AM