Google, Ask.com & Yahoo! Earth Day Logos | Main | Google AdSense (Publisher) Revenue Share Highest Ever - Maybe

April 23, 2007

A Robots.txt File Marked As Supplemental By Google?

A WebmasterWorld thread has someone reporting that he noticed a robots.txt file has been indexed and marked in the supplemental index at Google.com

It is clear that Google indexes robots.txt files. The thread creator wants to know what is means if a robots.txt file is in the Google supplemental index.

I agree with g1smd that "There are no advantages or disadvantages. It just is."

So leave it as is and let Google worry about it.

Forum discussion at WebmasterWorld.



Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!

posted rustybrick in Google Optimization at April 23, 2007 7:45 AM Comments (3)

• Google Co-op Cartoon Barry

Comments

wow... is HAL starting to get away from Google?

 

Posted by chris boggs at April 23, 2007 08:19

The robots.txt must have been linked from somewhere to get indexed. If it is not linked anymore, then it will fall into the supplemental index and then slowly fall out of the index. This is the normal behavior.

What puzzles me more is how you could go about getting the robots.txt blocked from being indexed. Can you list it in the robots.txt? :-) The same goes for XML or text sitemap files. I guess you can only drop the links and wait.

 

Posted by JohnMu at April 23, 2007 08:59

I agree with g1smd. :) Most likely, it means that someone did a hyperlink to that robots.txt. It doesn't hurt or help.

 

Posted by Matt Cutts at April 23, 2007 21:57

Post a comment (Note: Can Take 120 Seconds For Your Comment To Show Up)

Do you want us to save your personal Information?

Premium Sponsors + advertise

To subscribe to the Search Engine Roundtable, click here