Page load time (speed) is a factor currently in the AdWords quality score. But soon it may be coming to Google's organic ranking algorithm. If you have a really slow site, it may impact how high you rank in Google. That was the main news coming out of PubCon last week, minus the Caffeine launch.
It is currently not in the algorithm, according to Matt, but who knows - maybe they are testing this already. Matt was clear that Google wants the web to be a faster place and Google does control much of what people see on the web. So Google can influence that people find faster web pages, over slower ones.
You can hear Matt talk about this 2 minutes and 52 seconds into this video:
Google also has a tool to test page speed at http://code.google.com/speed/page-speed/ - so get ready.
I should add, Google has hundreds of ranking factors. Adding one more, depending on the weight they assign to it, shouldn't shuffle things up much for most sites. Just make sure your site loads fast - it is a good thing to have anyway.
Forum discussion at WebmasterWorld.