Bing Not Honoring Robots.txt Directives?

Sep 9, 2009 - 8:45 am 0 by
Filed Under Bing Search

Over the past few weeks, I have been noticing threads pop up in the Bing forums with complaints from webmasters that Bing's bot, aka MSNBot is not honoring their robots.txt directives.

It was not just one thread, but at least four. They include one from yesterday over here, one started on September 3rd over here, one from September 1 over here and one started on August 25th over here.

I ignored the first three, trying to give Microsoft the benefit of the doubt. Hoping it is a webmaster issue or someone spoofing MSNBot. But four threads on the same topic, all within a few weeks of each other does stand out as a possible issue.

I have personally not confirmed the issue, since I have no interest in blocking MSNBot from crawling any parts of my sites - but others don't like Bing as much as I do.

Forum discussion at Bing Forums here, here, here and over here.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Volatility, Helpful Content Update Gone, Dangerous Search Results & Ads Confusion - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google Will Fight Site Reputation Abuse Spam Both With Manual Actions & Algorithms

Apr 16, 2024 - 7:51 am

Google Search Tests Thumbs Up/Down Buttons In Product Grid Results

Apr 16, 2024 - 7:41 am
Google Ads

Google Ads "Similar Product" Carousel

Apr 16, 2024 - 7:31 am
Google Search Engine Optimization

Clarification: Google Search Supports Images Referenced From src Attribute

Apr 16, 2024 - 7:21 am
Google AdSense

Google AdSense New Ad Intents Formats - Links & Anchors In Content

Apr 16, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: April 15, 2024

Apr 15, 2024 - 4:00 pm
Previous Story: Google Webmaster Tools Keywords Report Showing Blank Lines?