Make Sure Your Robots.txt File is UTF-8

Jul 28, 2008 - 7:45 am 1 by

A Google Groups thread shows the tail of a webmaster who had issues with his robots.txt file. The robots.txt file was uploaded in what is called byte-order mark (BOM) encoding, which threw off Google, when trying to retrieve and understand the webmaster's robots.txt file.

Google Groups member, Phil Payne noticed the issue right away, by using The HTML editor this webmaster was using uploaded his robots.txt file in the BOM encoding. Google and other search engines prefer to see the robots.txt file in UTF-8 encoding.

Googler, JohnMu, confirmed the issue saying:

Phil was right on target there, it seems the BOM at the beginning of the file might be throwing us off. The easiest way to get around this issue is to have an empty line (or a comment) in the top of your robots.txt file -- that way it'll work even if you have a BOM in your file.

In short, the webmaster fixed the encoding issue by editing the file manually and reuploading.

Forum discussion at Google Groups.


Popular Categories

The Pulse of the search community


Search Video Recaps

Google Core Update Coming, Ranking Volatility, Bye Search Notes, AI Overviews, Ads & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 24, 2024

Jul 24, 2024 - 10:00 am
Google Updates

Google Search Ranking Algorithm Volatility - July 23rd

Jul 24, 2024 - 7:51 am
Google Search Engine Optimization

SEO Poll: 54% Of SEOs Saw Positive Effects From Google Updates

Jul 24, 2024 - 7:41 am
Google Ads

New Google Local Service Ads Budget Options

Jul 24, 2024 - 7:31 am
Google News

Google Ad Revenue Up 11% - $64.62 Billion In Ad Revenue

Jul 24, 2024 - 7:21 am

Google Outlines Videos In Search Results

Jul 24, 2024 - 7:11 am
Previous Story: Video Recap of Weekly Search Buzz :: July 27, 2008