Multiple Robots.txt Files for Single Domain

Apr 1, 2009 - 8:13 am 5 by

A HighRankings Forum thread asks why do some people use more than a single robots.txt file to control and instruct search spiders how to crawl and access their content. That is a good question. Typically, the spiders will only listen to the robots.txt file found in the root level. So technically, if you place a robots.txt on a subdomain, the search engine will likely ignore it. I do not believe the same applies to subdomains, where subdomains have their own root levels.

HighRankings administrator, Randy, said:

robots.txt anywhere but the Root level will be ignored by the spiders. In fact it would surprise me if it's ever even queried. robots.txt is not like .htaccess where you can control things on a per directory level.

The only way a subdirectory robots.txt might be valid is the rare case where someone has a domain name parked on a subdirectory of another domain. Or possibly if the subdirectory is really a subdomain, though that one too is questionable in my mind and isn't something I've tested to see if spiders look for a robots.txt for each subdomain.

I love what Ron Carnell added:

FWIW, I almost always back up a file before modifying it. My ex-wife always said I had trust issues? At any rate, I probably have a few copies of robots.txt laying around on more than a few sites. I don't worry about it because, as you pointed out, the only one that counts is in the root.

I believe Google often uses individual sitemaps per subdomain, to control their content.

Forum discussion at HighRankings Forum.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
Google Core Update Flux, AdSense Ad Intent, California Link Tax & More - YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google: We Need Very Few Links To Rank Pages; Links Are Less Important

Apr 22, 2024 - 7:51 am
Google

Google Tests Discussion and Forums Questions With Answers Box

Apr 22, 2024 - 7:41 am
Google Search Engine Optimization

Forbes Blocks Its Coupon Directory Before Google Enforces The Site Reputation Abuse Policy

Apr 22, 2024 - 7:31 am
Google Maps

Google Tests Posts Button In Local Listing Business Profile

Apr 22, 2024 - 7:21 am
Google Ads

Google Search With 16 Shopping Ads

Apr 22, 2024 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: April 19, 2024

Apr 19, 2024 - 4:00 pm
Previous Story: Google Maps Incredibly Slow? Troubleshoot Issues With Google