Branko at SEO Scientist wrote Anatomy of a Google Filter/Penalty (or how not to do nofollow PR sculpting). In that post, he goes through a specific client who saw signs of the minus X penalty (i.e. -6, -30, -60, -950 and so on).
Branko summarized the pattern he saw, and actions he took, via a chart:
The full size version is available at the post, but as you can see, he tried to isolate certain factors. His observations led to a few theories, including that the filter is not a complete ban of a site, it is country specific to the US, it appears to be keyword sensitive, and both on and off page factors seem to trigger it. Based on his observations, Branko offers several pieces of advice.
For more details on this, I recommend checking out his post.
Forum discussion at Sphinn.