Gary Illyes and Chery Prommawin released their second video in the Non-Canonical Cabinet series; this one is named What can you actually put in a robots.txt? Here they discuss what is, and is not, supported by robots.txt files. And while Google may support features or directives, other search engines or AI crawlers might not.
Here are the top level takeaways:
- Googlebot uses the robots.txt file to limit crawling
- Your robot.txt file should follow RFC 9309
- Use percent-encoding for non-English characters in rule paths
- And check out Google's document for more details
Here are some points I pulled out from listening to the video:
- Don't assume a robots.txt rule is working just because it looks understandable to a human. Validate the syntax and test the actual URL patterns.
- Be careful when blocking URLs containing international characters or emojis. Encoding inconsistencies can undermine a rule, particularly across different crawler implementations.
- Separate crawling controls from AI training policies. Blocking an AI crawler from accessing a page is different from telling a company not to train on content it has already obtained.
- Use the correct crawler name. AI crawler rules need to identify the intended crawler using its recognized user-agent token.
- Remember that robots.txt isn't a security mechanism. It requests that compliant crawlers refrain from accessing specified paths; it doesn't prevent access by unauthorized clients or guarantee that every bot will comply.
My favorite part of the video is when Cherry pulls out of the cabinet a framed photo of John Mueller and hands it to Gary:
Here is the full video:
Here is the transcript:
Gary Illyes and Cherry Prommawin Discuss robots.txt
Cherry Prommawin: Gary, Gary. The cabinet is calling to me.
Gary Illyes: The only thing calling to you is chaos. Cherry, leave the cabinet alone.
Gary Illyes: We are addressing a backlog from the deepest depths of our event Q&A. Every time someone asks us a weird question, it goes right in here.
Gary Illyes: We've accumulated so much technical baggage over the years. If a fire inspector walks through that door, I don't know you, Cherry.
Gary Illyes: And it all lives right in here in this non-canonical cabinet. Behold.
Cherry Prommawin: A no-entry sign in a filing cabinet. Are we worried people are going to climb inside?
Gary Illyes: Oh, Cherry, you can never be too careful.
Gary Illyes: The sign is here because today, we are talking about the gatekeeper of your website, the robots.txt file.
Cherry Prommawin: The love of my life.
Gary Illyes: Robots.txt, that file is just a simple, plain-text file that lives at the root of a website to tell automated bots what they can and cannot do.
Cherry Prommawin: So simple. So beautiful.
Gary Illyes: The rules apply to search engine crawlers, AI training crawlers, and other well-behaved crawlers.
Cherry Prommawin: It's like a set of house rules for crawlers.
Gary Illyes: Before Googlebot or any other well-behaved crawler even looks at the content on a site, it checks this file first to see which rooms it's allowed to enter and which ones are strictly off-limits.
Gary Illyes: And this is all defined by an internet standard, RFC 9309.
Cherry Prommawin: So beautiful.
Gary Illyes: Oh, okay.
Cherry Prommawin: What people want to know is how weird can you actually make this file? Can we make it pretty? Can we use non-English characters? Can we tell AI bots to stay out using a literal stop-sign emoji?
Gary Illyes: Seriously? Okay, let's go to the desk.
Gary Illyes: A robots.txt file is not your personal art canvas.
Cherry Prommawin: It's a plain-text file.
Gary Illyes: It is the very first thing many crawlers look at when they visit your site, to see where they are allowed to go.
Gary Illyes: They only care about things they can actually read.
Cherry Prommawin: Okay, can I at least make fun ASCII art in the comments, like a giant ASCII chicken guarding my site?
Gary Illyes: Technically, yes.
Gary Illyes: Start your lines with a hash and crawlers will completely ignore them.
Gary Illyes: So while it's a questionable life choice, you can paste a giant ASCII chicken in there.
Cherry Prommawin: I once found a website whose robots.txt file was 97% script and only 3% directives.
Gary Illyes: Did it break a parser?
Cherry Prommawin: No, waste bandwidth.
Gary Illyes: Yeah, Cherry.
Cherry Prommawin: Isn't that your website?
Gary Illyes: I have many websites. Okay, hey chicken, drawing matters.
Gary Illyes: But if you accidentally mess up the formatting, that might be dangerous. As we said, the standard is RFC 9309.
Gary Illyes: Google follows that standard. The parser reads line by line. It looks for specific rules: user-agent, allow, disallow, and sitemap.
Gary Illyes: If you mess up a line or try to get fancy with your spacing, the system will just silently discard the line it doesn't understand and move on.
Gary Illyes: It's a simple machine. It only knows what to do with a few commands.
Cherry Prommawin: Yeah, like you said, four of them.
Gary Illyes: If you write "don't crawl secret page," Google looks at "don't crawl," realizes it's not a valid rule, ignores it, and might actually crawl your secret page anyway.
Cherry Prommawin: What if there's a crawler that's using the chicken emoji as a product token? Can I write user-agent chicken emoji?
Gary Illyes: Does the chicken bot stop?
Cherry Prommawin: There's a chicken bot?
Gary Illyes: Sadly, no.
Gary Illyes: Well, but even if there were, user-agent tokens can only have letters and dashes.
Gary Illyes: An emoji is a multibyte UTF-8 character, and I don't know any encoding that would fit that letter-dash requirement.
Gary Illyes: So it just won't work at all.
Cherry Prommawin: Wait, does that mean we also cannot have non-English characters in there?
Gary Illyes: No chicken emoji or other non-English characters in the user-agent token.
Cherry Prommawin: Chicken made me think.
Gary Illyes: What do you think happens when site owners use chicken or some localized text in their URL paths?
Cherry Prommawin: Oh, you mean non-English characters. Maybe German umlauts, the Portuguese cedilla? Letters with little hats and tails.
Gary Illyes: Yeah, also maybe an entirely different set of alphabets, like Arabic, Hindi, Japanese, or Chinese.
Gary Illyes: It's actually the exact same concept for emojis and non-English characters.
Gary Illyes: Take a path like a German chicken page or a full Chinese-character path to a delicious fried-chicken page, like henhouse.
Cherry Prommawin: Your pronunciation is surprisingly decent, Gary. You only speak in server logs and coffee. Do you even know what I just said?
Gary Illyes: Please, I know the cadence of the best fried chicken ever in any language. But tell me more about the parser.
Cherry Prommawin: Okay, Googlebot handles international scripts and chicken like any HTTP client does behind the scenes. Those characters are multibyte UTF-8.
Cherry Prommawin: Google's parser is flexible and will automatically translate Chinese characters, German umlauts, or emojis by encoding them.
Cherry Prommawin: Actually, it's the same thing Google does when crawling for stuff, but other or older crawlers on the web might get confused. To be safe across the entire global internet infrastructure, you should use the encoded version in your robots.txt rules.
Gary Illyes: Say you want to block crawlers from visiting—
Cherry Prommawin: Get you a chicken emoji.
Gary Illyes: Here, mirror what your server expects and actually write its percent-encoded form.
Cherry Prommawin: Keep it standard so every bot that speaks robots.txt respects your boundaries. If they can't process something like a raw chicken, they might just ignore the whole disallow line and crawl everything you try to protect.
Gary Illyes: I have absolutely nothing to add to that.
Gary Illyes: Okay, let's talk about the big one: generative AI.
Cherry Prommawin: Everyone wants to know how to talk to AI crawlers. Can I use a robots.txt file to tell AI models, "Hey, you can crawl my site, but don't use my chicken diary to train your generative models"?
Gary Illyes: That is a good topic.
Gary Illyes: So, as defined by RFC 9309, robots.txt is an access-preference protocol. It only tells bots, "You can crawl this page," or, "You shouldn't crawl this other page."
Gary Illyes: It doesn't dictate what a system does with the data after it has already crawled it.
Gary Illyes: So disallow bluntly means, "Do not enter the room." It has nothing to do with looking at my chicken diary or putting it in a database.
Cherry Prommawin: Shanghai a saucepan?
Gary Illyes: Yeah, if you let an AI bot crawl your site, it processes the robots.txt as a machine-readable file, not as an arbitrary plain-text file.
Gary Illyes: If you want to stop a specific AI crawler from accessing your pages entirely, you have to target its specific user-agent token explicitly in the file.
Gary Illyes: You can't write a fancy custom rule like "no-I-chicken-chicken-diary." The bot's parser will just throw it in the trash, character by character.
Gary Illyes: Time for the recap.
Cherry Prommawin: Do you want to go first?
Gary Illyes: First, well-behaved crawlers, including Googlebot, use the robots.txt file to limit crawling.
Gary Illyes: Second, your robots.txt file should follow the official internet standard, RFC 9309.
Gary Illyes: And finally, use percent-encoding for emojis and non-English characters in robots.txt rule paths to keep them safe for all crawlers.
Cherry Prommawin: Finally, finally, if you want to get to know robots.txt better, go to the two links in the description, and please keep your ASCII art out of critical directives.
Gary Illyes: Let's keep it boring. Beautifully boring.
Cherry Prommawin: If you have a snippet of weird source code, a strange rule, or an absolute mystery folder you want us to pull from the non-canonical cabinet, drop a comment below.
Gary Illyes: Or don't. We're not telling you how to live your lives.
Cherry Prommawin: Next episode, we are going to talk about 404. Subscribe for more.
Gary Illyes: There cannot possibly be more.
Cherry Prommawin: Oh.
Forum discussion at LinkedIn.
