Gary Illyes from Google said that XML Sitemap files do not directly impact your rankings in Google Search. This is not new news but I like how he said "No, not direct anyway" to answer the question.
Here are those tweets where the question asked was "Gary, is there a problem/ranking disadvantage not using a sitemap.xml at all?"
No, not direct anyway
— Gary "鯨理" Illyes (@methode) June 14, 2019
We know Google said removing XML Sitemaps doesn't hurt a site and changing the XML Sitemaps location also shouldn't hurt one's rankings. So not having a sitemap file also won't hurt.
XML Sitemaps help Google discover the URLs on your web site. Google ranks those URLs but the XML Sitemap file is about Google discovering those URLs, not directly ranking those URLs. Crawling, indexing and ranking are different things.
Forum discussion at Twitter.
