Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler

Feb 13, 2008 - 7:28 am 1 by
Filed Under Bing Search

The Live Search Blog announced several updated to their crawler. The first is a name change to reflect the upgrade, previously named msnbot/1.0, it is now named msnbot/1.1.

The bulk of the changes include the HTTP Compression and Conditional Get support.

HTTP compression is supported by many other bots already and is now supported by Microsoft. So for a more efficient crawl, Microsoft now supports gzip and deflate compression methods.

The Conditional Get will aid the crawler in deciding if it should crawl your page again. If you haven't made a change to your page, then crawling the page would not be too efficient. So Microsoft's crawler will include the "If-Modified-Since" header & time of last download in the GET request and when available, the crawler will include the "If-None-Match" header and the ETag value in the GET request. If content has not changed, the web server will respond with a 304 HTTP response.

Sebastian has a Q&A with Nathan Buggia from Microsoft on these changes and Vanessa Fox has a look at how other search engines handle it.

Forum discussion at WebmasterWorld.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Ranking Volatility, Tons Of Google Bugs, Most Read Articles, Bing Deep Search & Quality... - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 1, 2024

Mar 1, 2024 - 4:00 pm
Search Video Recaps

Search News Buzz Video Recap: Google Ranking Volatility, Tons Of Google Bugs, Most Read Articles, Bing Deep Search & Quality & SEOs Not Loving Google

Mar 1, 2024 - 8:01 am

Google Court Document Says Search Quality Continues To Improve

Mar 1, 2024 - 7:51 am
Google Search Engine Optimization

Google Structured Data Carousels Beta Documentation Added

Mar 1, 2024 - 7:41 am

Now You Can Opt Out Of Google Search Place Entities In Page Insights

Mar 1, 2024 - 7:31 am
Google Maps

Google Updates Business Profiles Customer Interactions Every Few Days

Mar 1, 2024 - 7:21 am
Previous Story: Daily Search Forum Recap: February 12, 2008