An excellent WebmasterWorld thread has one webmaster who noticed he was accidentally linking to thousands of 404ed internal pages. He believes due to that issue, his site's rankings have degraded in Google.
To me, it doesn't sound far-fetched. I wouldn't call this a penalty, but rather a major internal linking issue. Internal linking is important for many reasons, two of which include giving search engines a way to discover your content and showing them what your content is about using anchor text. If you link to pages that don't exist, well - that cannot help you or the search engine. Forget the importance of using internal linking for other reasons, discovery is the basic purpose.
PageOnResults in the forum also added his personal experience with such an issue:
I have personally experienced a degradation in rankings due to a 404 mishap. The site has approximately 1,000 documents. After launching a new application, we discovered a bug in the rewrite that caused Googlebot to index 1,000+ documents that were returning a 404. Unfortunately we did not catch this in a timely manner and sure enough, not long after, we started to lose organic traffic.Since we are in a very controlled environment, we were able to determine that the loss in organic traffic was a direct result of the 404 issues. After correcting the rewrite, traffic returned to normal within 30 days and actually improved a little thereafter.
It happens, and you can recover.
Forum discussion at WebmasterWorld.
