Google will partially remove a penalty when they see you take significant action on a notification.
John Mueller from Google said in response to a Google Webmaster Help thread that they do indeed do this. One webmaster received a notification that the inorganic link penalty they received was partially revoked. Google said they were able to "partially revoke a manual action" taken against the site.
Here is the full message:
Thank you for your request.
After re-evaluating your site's backlinks we are able to partially revoke a manual action. There are still inorganic links pointing to your site that we have taken action on. Once you've been able to make further progress in getting these links removed, please reply to this email with the details of your clean-up effort.
It is nice to see Google not taking an all or nothing approach with these penalties.
John from Google gave a bit more context for these partial revocations:
That usually means that the team has been able to remove a part of the manual actions being taken due to the changes that you've made. It sounds like there still are some issues that you might want to review & resolve though. Generally speaking, it can take a bit of time for these kinds of changes to bubble up, and to be visible in search results, it would be rare to see a jump right afterwards. My recommendation (not knowing the specific case/site) would be to follow the advice of the search quality team and to continue working on removing any unnatural links that your site may have collected over time.
Just a reminder, those penalty notifications you get via webmaster tools, those are manually sent.
Forum discussion at Google Webmaster Help.
Image credit to ShutterStock for referee flag