Google, Ask.com & Yahoo! Earth Day Logos | Main | Google AdSense (Publisher) Revenue Share Highest Ever - Maybe

A Robots.txt File Marked As Supplemental By Google?

A WebmasterWorld thread has someone reporting that he noticed a robots.txt file has been indexed and marked in the supplemental index at Google.com

It is clear that Google indexes robots.txt files. The thread creator wants to know what is means if a robots.txt file is in the Google supplemental index.

I agree with g1smd that "There are no advantages or disadvantages. It just is."

So leave it as is and let Google worry about it.

Forum discussion at WebmasterWorld.



Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!

posted rustybrick in Google Optimization at April 23, 2007 7:45 AM Comments (3)

Comments

wow... is HAL starting to get away from Google?

 

The robots.txt must have been linked from somewhere to get indexed. If it is not linked anymore, then it will fall into the supplemental index and then slowly fall out of the index. This is the normal behavior.

What puzzles me more is how you could go about getting the robots.txt blocked from being indexed. Can you list it in the robots.txt? :-) The same goes for XML or text sitemap files. I guess you can only drop the links and wait.

 

I agree with g1smd. :) Most likely, it means that someone did a hyperlink to that robots.txt. It doesn't hurt or help.

 

Post a comment (Note: Can Take 120 Seconds For Your Comment To Show Up)

Do you want us to save your personal Information?


To subscribe to the Search Engine Roundtable, click here