A Google Groups thread shows the tail of a webmaster who had issues with his robots.txt file. The robots.txt file was uploaded in what is called byte-order mark (BOM) encoding, which threw off Google, when trying to retrieve and understand the webmaster's robots.txt file.

Google Groups member, Phil Payne noticed the issue right away, by using rexswain.com/httpview.html. The HTML editor this webmaster was using uploaded his robots.txt file in the BOM encoding. Google and other search engines prefer to see the robots.txt file in UTF-8 encoding.

Googler, JohnMu, confirmed the issue saying:

Phil was right on target there, it seems the BOM at the beginning of the file might be throwing us off. The easiest way to get around this issue is to have an empty line (or a comment) in the top of your robots.txt file -- that way it'll work even if you have a BOM in your file.

In short, the webmaster fixed the encoding issue by editing the file manually and reuploading.

Forum discussion at Google Groups.

Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!