Will Google Crawl Your Site Without a Robots.txt File? It Depends

Jun 18, 2008 - 7:26 am 4 by

I found a very interesting tidbit from a Google Groups thread on unreachable robots.txt files.

I always believed that a site does not need a robots.txt file. In fact, this site does not have a robots.txt file and yet we are very well indexed. Proof that you don't need a robots.txt file to allow Google to index your site. Right?

Well, maybe not. Googler, JohnMu, said in the Google Groups thread that if your robots.txt file is unreachable due to timing out or other issues, not including a 404 not found status, Google "tends not to crawl the site at all just to be safe."

You hear that? Google might not crawl your entire site if it cannot reach your robots.txt file properly.

In the case in the thread, the robots.txt file was unreachable due to a complex set of redirects that made Googlebot very dizzy.

John explains later on that "unparsable robots.txt" files are "generally" okay, since Google is getting back some type of server response. When you have an issue is when generally "the URL is just unreachable (perhaps a "security update" that ended up blocking us in general) or situations like this where we give up trying to access the URL (which in a way is unreachable as well)," said John.

So, for those picky Belgium's, just make your robots.txt file unreachable, and there you go. :)

Forum discussion at Google Groups.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: July 4, 2025

Jul 4, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google June 2025 Core Update, Search Volatility, Insights Report, Ads & More

Jul 4, 2025 - 8:01 am
Bing Search

Bing Search Tests Zoomable & Sticky Related Searches

Jul 4, 2025 - 7:51 am
Google

Google AI Mode Can Respond In Non-English Languages

Jul 4, 2025 - 7:41 am
Bing Search

Bing Tests Local Place Listings In Green

Jul 4, 2025 - 7:31 am
Google Ads

Google Merchant Center Gains Automatic Shipping Updates

Jul 4, 2025 - 7:21 am
Previous Story: Daily Search Forum Recap: June 17, 2008