(3)
A WebmasterWorld thread has someone reporting that he noticed a robots.txt file has been indexed and marked in the supplemental index at Google.com
It is clear that Google indexes robots.txt files. The thread creator wants to know what is means if a robots.txt file is in the Google supplemental index.
I agree with g1smd that "There are no advantages or disadvantages. It just is."
So leave it as is and let Google worry about it.
Forum discussion at WebmasterWorld.
Comments:
chris boggs
04/23/2007 12:19 pm
wow... is HAL starting to get away from Google?
JohnMu
04/23/2007 12:59 pm
The robots.txt must have been linked from somewhere to get indexed. If it is not linked anymore, then it will fall into the supplemental index and then slowly fall out of the index. This is the normal behavior. What puzzles me more is how you could go about getting the robots.txt blocked from being indexed. Can you list it in the robots.txt? :-) The same goes for XML or text sitemap files. I guess you can only drop the links and wait.
Matt Cutts
04/24/2007 01:57 am
I agree with g1smd. :) Most likely, it means that someone did a hyperlink to that robots.txt. It doesn't hurt or help.