(1)
A WebmasterWorld thread has one webmaster complaining that Microsoft is hiding the user agent for the spider, BingBot.
The webmaster said:
I'm not really sure if this is regular practice with MSN (Bing) or not. Since a few weeks back, they've been spidering my website without a proper User Agent (meaning their UA does NOT say anything about bingbot. It's just a regular browser UA). This happens from three different IP ranges. Yes, I've checked thoroughly and the IP rangers are registered under MSN.I doubt this is manual review, since it doesn't look at all as normal human behavior.
Did they officially stop declaring their bots in User Agent strings?
So either Microsoft has a weird bug, someone is spoofing the crawler in a weird way or Microsoft is doing this on purpose?
I should note, I did not check my log files to verify this on my sites. So I am going with "zerillos" post and with no back up evidence.
However, Microsoft is known for their crawling issues. Most recently, crawling too hard, updating IPs and many more. BingBot is pretty new, coming to us just over a year ago.
Forum discussion at WebmasterWorld.
