It is interesting how the major four tend to discuss similar topics at the same time. Yesterday, I wrote on the topic of Ask Jeeves's Crawl Delay, where the Ask Jeeves rep clarified some of the rumors out there on how Ask handles it. Late last night, Yahoo! posts an entry at the Yahoo! Blog named Yahoo! Search Tips for Webmasters: Saving Bandwidth. One of the tips provided included using Yahoo!'s Crawl-Delay. In fact, Yahoo! constantly reminds Webmasters to use this parameter as the conferences. I believe the only other engine that made a big point of this was MSN.
Other tips: - Gzipped Files - did this, but the day after I had a major sever crash, probably unrelated but I am done with this method. I rather pay for more bandwidth then try it again. - Smart Caching - we do this all the time, it works great for many reasons, not only for saving bandwidth.
