Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009 - 8:38 am 1 by
Filed Under Bing Search

Here is one more oddity to add to Microsoft Bing's web crawler, MSNBot. Why on earth are people reporting that MSNBot is crawling the same page twice, once for the compressed version and then once again for the uncompressed version? Technically, it should probably only crawl once and it should opt for the compressed, gzip version - don't you think?

We have two threads complaining about this, one oldish one at WebmasterWorld and another at Bing Forums. Let me quote the Bing thread:

I've notice that bing is crawling each page of my website twice, first making an HTTP 1.1 request and getting a compressed response then immediately issuing an HTTP 1.0 request to receive the same page without gzip compression The following lines from my log show the issue (there are thousands more similar occurrences): - - [13/Dec/2009:14:58:42 +0000] "GET /specimen/235698/ HTTP/1.1" 200 1742 "-" "msnbot/2.0b (+" - - [13/Dec/2009:14:59:06 +0000] "GET /specimen/235698/ HTTP/1.0" 200 4259 "-" "msnbot/2.0b (+" - - [13/Dec/2009:15:03:08 +0000] "GET /specimen/250262/ HTTP/1.1" 200 1733 "-" "msnbot/2.0b (+" - - [13/Dec/2009:15:03:14 +0000] "GET /specimen/250262/ HTTP/1.0" 200 4164 "-" "msnbot/2.0b (+" This seems a waste of bandwidth and completely defeats the point of supporting http compression.

Indeed a waste of bandwidth and yes, it defeats the point of supporting HTTP compression.

A Bing representative, Brett Yount said:

could you please mail this information to [email protected] and I will get our crawling team to check it out?

But we have no confirmation from Bing on why this issue is occurring or when it will be fixed. Like I said, just one more oddity to add to MSNBot's crawl behavior.

Forum discussion at WebmasterWorld and Bing Forums.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Weekend Volatility, Google On Search Leak, Elizabeth Tucker Interview & Apple Intelligence - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: June 17, 2024

Jun 17, 2024 - 10:00 am
Google Search Engine Optimization

The Maddening Adventure Of Tracking AI Overviews In Google Search Console

Jun 17, 2024 - 7:51 am
Google Ads

Google Ads Forcing Some Advertisers Off Credit Card Payments

Jun 17, 2024 - 7:41 am
Apple App Store

Apple Updates Applebot Docs With Applebot-Extended, Reverse DNS & More

Jun 17, 2024 - 7:31 am
Google Maps

Google Maps Updates Fake Engagement Policy With More Detail

Jun 17, 2024 - 7:21 am
Google Search Engine Optimization

Google Updates hreflang Documentation While I Was Offline

Jun 17, 2024 - 7:11 am
Previous Story: When Will Google Fix the Web Design Local Pack Bug?