Is Microsoft Live Search Crawling More But Indexing Less?

Nov 24, 2008 - 7:42 am 4 by
Filed Under Bing Search

I spotted two thread this month discussing how little Microsoft is indexing their sites but how often Microsoft's bot, MSNbot, is crawling their sites.

A WebmasterWorld thread and DigitalPoint Forums thread has details of the newish behavior from MSNbot. BillyS at WebmasterWorld explains:

I was just looking through my logs and noticed that msnbot was crawling our site pretty hard, grabbing about 10% of the site in the last half hour or so.

I just checked the site: command on Live and we've only got about 100 pages in their index now - which is fewer than the number of pages mentioned above.

Billy, as others, wonder if they should just block MSNBot all together, since they feel the traffic they received from Live Search is not worth the stress the bot puts on the server when they crawl.

Let's do some comparisons of Google versus Live Search in site command counts:

So either I am doing something wrong or Live Search's site command is wrong, or Live Search forgot how to index pages?

Forum discussion at WebmasterWorld and DigitalPoint Forums.

Update: Microsoft sent me a response to this post, which I felt would be great to add.

For webmasters, It is problematic to use the “site:” operator to determine how many pages for a site are included in the Live Search index. The “Site:” operator generates an estimate of the pages in the index. These numbers can vary wildly depending on when you execute the query.

You posed the question about whether users should block MSNbot because traffic from the bot is not worth the stress on your servers. Obviously, we would prefer that customers not block MSNbot, rather customers who are concerned with stress from Live Search crawls should add the crawl-delay parameter to their robots.txt file. This can help reduce the load on your servers and still be a part of the Live Search results. Webmasters can refer to the MSNBot support page for more information on crawl-delay.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Search Ranking Update, Recipe Blogs Drop, Google Hits Reviews & More SEO, PPC and Local - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google: Sites Wrongfully Using The Google Indexing API Often Are Spammy & Low Quality

Feb 20, 2024 - 7:51 am
Google Search Engine Optimization

Google Merchant Center Updates Estimated Delivery Times Calculation

Feb 20, 2024 - 7:41 am
Google Search Engine Optimization

Google Merchant Center Requires Meta Labels On AI Generated Images

Feb 20, 2024 - 7:31 am
Google Maps

Google Local Pack Tests Restaurant Price Ranges

Feb 20, 2024 - 7:21 am

Google Search Can Wait On Customer Service Hold For You

Feb 20, 2024 - 7:11 am
Google Search Engine Optimization

New Google Search Support For Product Variant Structured Data

Feb 20, 2024 - 6:00 am
Previous Story: Was There a November Yahoo Update?