Make Sure Your Robots.txt File is UTF-8

Jul 28, 2008 - 7:45 am 1 by

A Google Groups thread shows the tail of a webmaster who had issues with his robots.txt file. The robots.txt file was uploaded in what is called byte-order mark (BOM) encoding, which threw off Google, when trying to retrieve and understand the webmaster's robots.txt file.

Google Groups member, Phil Payne noticed the issue right away, by using rexswain.com/httpview.html. The HTML editor this webmaster was using uploaded his robots.txt file in the BOM encoding. Google and other search engines prefer to see the robots.txt file in UTF-8 encoding.

Googler, JohnMu, confirmed the issue saying:

Phil was right on target there, it seems the BOM at the beginning of the file might be throwing us off. The easiest way to get around this issue is to have an empty line (or a comment) in the top of your robots.txt file -- that way it'll work even if you have a BOM in your file.

In short, the webmaster fixed the encoding issue by editing the file manually and reuploading.

Forum discussion at Google Groups.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: August 15, 2025

Aug 15, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Volatility Continues, Preferred Sources, Site Hack Demolishes Traffic & Google On AI

Aug 15, 2025 - 8:01 am
Google

Chartbeat: Traffic From Google To Publishers Stable Over Years

Aug 15, 2025 - 7:51 am
Search Engine Optimization

Google's John Mueller Suggests AI SEO Acronyms May Lead To Scams & Spam

Aug 15, 2025 - 7:41 am
Google

Google Search Age Verification: Asking To Confirm You're An Adult

Aug 15, 2025 - 7:31 am
Google

Google Looking Into Adding Analytics For Preferred Sources

Aug 15, 2025 - 7:21 am
Previous Story: Video Recap of Weekly Search Buzz :: July 27, 2008