Google: It Is A Bad Sign If Your Robots.txt Or Sitemap File Is Ranking For Normal Queries

Nov 8, 2019 - 7:47 am 0 by

Bad Apple

We had a slew of tweets triggered by Gary Illyes of Google and then followed up by John Mueller of Google around robots.txt and XML sitemap files ranking in Google. In short, if they rank for normal queries, John Mueller said "that's usually a sign that your site is really bad off and should be improved instead."

Let's start with Gary's tweet:

A robots.txt file can be indexed and ranked in Google is what he is saying.

John then adds that you can block these from being indexed using the x-robots-tag HTTP header.

But if you do see your robots.txt file ranking or your sitemap file ranking, he said " if your robots.txt or sitemap file is ranking for normal queries (not site:), that's usually a sign that your site is really bad off and should be improved instead."

You can also use the disallow John added:

Maybe I am misunderstanding, but John said disallow doesn't work here in 2018? Here is his tweet from back then?

I guess when you disallow in robots.txt it is too late anyway.

John said there is really no reason to let Google index your sitemap file, Google processes that differently:

Anyway, I thought you'd find these tweeted, compiled in one post, useful.

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 21, 2025

Mar 21, 2025 - 10:00 am
Search Engine Roundtable Vlog

Search News Buzz Video Recap: Google Core Updates, AI Search Leak, AI Overviews linking to Google, Assistant To Die, Bing News & More

Mar 21, 2025 - 8:01 am
Google Updates

Google March 2025 Core Update Volatility Bursts Or Tools Confused?

Mar 21, 2025 - 7:55 am
Google Ads

Google Ads Now Lets You Quickly Hide All Recommendations

Mar 21, 2025 - 7:51 am
Google Search Engine Optimization

Google: Stop Over Focusing On Tiny SEO Metrics

Mar 21, 2025 - 7:41 am
Google Search Engine Optimization

Google Search Console API To Support 24-Hour Hourly View

Mar 21, 2025 - 7:31 am
Previous Story: Google: Do Not Use Robots.txt To Block Indexing Of URLs With Parameters