Google: Robots.txt Can Be In Any Content Type

Jun 2, 2020 - 7:55 am 0 by

Lines Google

John Mueller of Google said that your robots.txt content-type can be anything really. He said "any content type is fine," adding "the important part is the content itself."

He also said he has "seen html pages that included valid robots txt lines among a valid html page."

If you are concerned, test it with Google's robots.txt validator.

Oh, the Penguin runner asked this question:

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: November 6, 2025

Nov 6, 2025 - 10:00 am
Google

Google Drops Support For More Structured Data Types & Search Features

Nov 6, 2025 - 7:51 am
Google

Google Search Tests Tinted Colored Background

Nov 6, 2025 - 7:41 am
Google Ads

Google Ads Fake Info For Advertiser Verification Programs Disallowed

Nov 6, 2025 - 7:31 am
Google

Google Definitions Box Tests Swapping In AI Overviews

Nov 6, 2025 - 7:21 am
Google Ads

Google Ads Editor Version 2.11 Is Now Available

Nov 6, 2025 - 7:15 am
 
Previous Story: Google & Bing Black Lives Matter Statements