A WebmasterWorld thread has one webmaster who is interested in just removing their XML sitemap completely from Webmaster tools. He is concerned that by doing so, his rankings will be impacted and is having second thoughts about removing the sitemap.
First, his reason for removing the sitemap file seems concerning to me but that is secondary to this topic. He wants to remove the sitemap because he is adding a forum and thus "won't be able to keep up with the sitemap." So he manages his sitemap manually? Really?
Anyway, if he does remove it, will it impact his rankings? Most likely not.
Googler, John Mueller, told us in May 2010 that removing URLs from a sitemap won't hurt your rankings. He said:
We use Sitemap files for discovering new URLs and recognizing when existing ones have changed; we don't remove URLs from the index that aren't in a Sitemap file.
So removing a sitemap won't necessarily hurt pages that are ranking well.
Moderator, GoodROI added:
I have rarely seen adding or removing sitemaps to dramatically impact a website. Just make sure your internal link structure is good and the site has no big technical issues.
You may want to keep the sitemap as a way to gather intelligence on which pages Google has indexed via Webmaster Central.
Forum discussion at WebmasterWorld.