Tedster started an outstanding thread at WebmasterWorld, something I personally never thought about but is all so valid.
All this stuff around having fast sites, cause Google is now looking at site performance and speed is important. It is not as important as most SEOs make it out to be, but the bottom line is, make sure your site is not too slow.
Tedster points out there is often confusion on how to determine if Google thinks your site is slow. There are two key reports in the Google Webmaster Tools:
(1) Crawl stats under Diagnostics section
(2) Site performance under Labs section
Crawl stats has a metric that tells you the time spent downloading a page (in milliseconds). That is, how much time GoogleBot spent downloading your content. It really doesn't say if it is fast or slow. I should note, you can set your crawl speed under Site configuration and Settings. If you can't, then maybe that is a sign of a slow site?
Site performance shows you data based on users with the Google Toolbar. Basically, it shows you how long a normal user takes to download the content on your site. I believe this is a more important metric for the site speed factor.
Site site performance is more about the speed algorithm factor, whereas, if you have a problem with GoogleBot crawling your site, your pages won't be included.
In any event, there is a lot of discussion around the two metrics and what you can do to leverage both at WebmasterWorld. So join the discussion.
Forum discussion at WebmasterWorld.