MSNbot Adds Method For Verification

Nov 30, 2006 - 8:17 am 0 by
Filed Under Bing Search

Microsoft has jumped on the bandwagon for allowing you to verify if the useragent, MSNBot, that is crawling your site, is truly from Microsoft or being spoofed by some content scraper. Google has released information in the past for verifying Googlebot and it is excellent to see Microsoft doing the same.

To catch fake MSNBots you can do as follows:

  1. When you get a page view request, it specifies a user-agent and an IP address. As I described above, all requests from Live Search use a user agent starting with the word ‘MSNBot’.
  2. If you see the MSNBot user-agent, it’s time to check the identity of the bot. Starting with the IP address (i.e. 207.46.98.149), you can use reverse DNS lookup to find out the registered name of the machine.
  3. Once you have the host name (in this case, livebot-207-46-98-149.search.live.com), you can check that it really is coming from Live Search. The name of all live search crawlers will end with ‘search.live.com’. If the name doesn’t end with ‘search.live.com’, you know it’s not really our crawler.
  4. Finally, you need to verify that the name is accurate. In order to do this, you can use Forward DNS to see the IP address associated with the host name. This should match the IP address you used in Step 2 – if it doesn’t, it means the name was fake.

More details at the Live Search Blog.

Forum discussion at WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search News

The Industry Mourns The Loss Of Jill Whalen - Content SEO Hero & Legend

Jun 22, 2025 - 1:20 pm
Search Forum Recap

Daily Search Forum Recap: June 20, 2025

Jun 20, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Search Volatility, AI Mode in Search Console, Audio Overviews and ChatGPT Search Quality

Jun 20, 2025 - 8:01 am
Google Search Engine Optimization

Google: Larger & Longer Issues Can Take Longer To Reverse In Google Search

Jun 20, 2025 - 7:51 am
Google Search Engine Optimization

Google Troubleshooting On Blocking Content From AI Mode & AI Overviews

Jun 20, 2025 - 7:41 am
Google

Google Video Search Tests Speaker Icon With Language Label

Jun 20, 2025 - 7:31 am
Previous Story: Google Cache Archiving More of Your Page?