John Mueller from Google said on Reddit that if your cache link in Google returns a 404 error, that it doesn't mean your rankings will be hurt in Google. John said "that can happen, it doesn't mean anything."
So if you are worried that this might result in your page being deindexed or the rankings may drop, Google says no, no need to worry about that. The Google Cache 404ing is an internal issue with the Google Cache server and doesn't reflect on indexing or ranking of the web page.
Google's John Mueller also said in the past that pages that do not have a cache link are not by default considered lower quality.
Forum discussion at Reddit.
