Is Microsoft Live Search Crawling More But Indexing Less?

Nov 24, 2008 - 7:42 am 4 by
Filed Under Bing Search

I spotted two thread this month discussing how little Microsoft is indexing their sites but how often Microsoft's bot, MSNbot, is crawling their sites.

A WebmasterWorld thread and DigitalPoint Forums thread has details of the newish behavior from MSNbot. BillyS at WebmasterWorld explains:

I was just looking through my logs and noticed that msnbot was crawling our site pretty hard, grabbing about 10% of the site in the last half hour or so.

I just checked the site: command on Live and we've only got about 100 pages in their index now - which is fewer than the number of pages mentioned above.

Billy, as others, wonder if they should just block MSNBot all together, since they feel the traffic they received from Live Search is not worth the stress the bot puts on the server when they crawl.

Let's do some comparisons of Google versus Live Search in site command counts:

So either I am doing something wrong or Live Search's site command is wrong, or Live Search forgot how to index pages?

Forum discussion at WebmasterWorld and DigitalPoint Forums.

Update: Microsoft sent me a response to this post, which I felt would be great to add.

For webmasters, It is problematic to use the “site:” operator to determine how many pages for a site are included in the Live Search index. The “Site:” operator generates an estimate of the pages in the index. These numbers can vary wildly depending on when you execute the query.

You posed the question about whether users should block MSNbot because traffic from the bot is not worth the stress on your servers. Obviously, we would prefer that customers not block MSNbot, rather customers who are concerned with stress from Live Search crawls should add the crawl-delay parameter to their robots.txt file. This can help reduce the load on your servers and still be a part of the Live Search results. Webmasters can refer to the MSNBot support page for more information on crawl-delay.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Rumbling, Manual Actions FAQs, Core Web Vitals Updates, AI, Bing, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 18, 2024

Mar 18, 2024 - 4:00 pm
Google Updates

Google Urges Patience As The March 2024 Core Update Continues To Rollout

Mar 18, 2024 - 7:51 am
Google

Official: Google Replaces Perspective Filter With Forums Filter

Mar 18, 2024 - 7:41 am
Google Maps

Google Business Profiles Now Offers Additional Review After Appeal Is Denied

Mar 18, 2024 - 7:31 am
Google Maps

EU Searchers Complaining About Google Maps Features Changes Related To DMA

Mar 18, 2024 - 7:21 am
Google

Google Showing Fewer Sitelinks Within Search

Mar 18, 2024 - 7:11 am
Previous Story: Was There a November Yahoo Update?