Robots.txt file ignored by Google?

Feb 20, 2006 - 10:49 am 5 — by
Filed Under Misc Google

Apparently the Googlebot is failing to heed the "Keep Away!" that the robots.txt file is supposed to yell authoritatively. Rand mentioned something like this the other day, and critter over at SEW forums described the following when he started a thread asking "What's The Point of A Robots.txt File If Google Ignores It?"

I noticed today Google indexing my images folder, even though I explicity prevent ALL SEARCH ENGINE SPIDERS from indexing that folder from various reasons. I have had this robots.txt file in the root of my site since the day it was launched and am quite annoyed and frustrated with Google for ignoring it and indexing the contents of the folder anyways.

Maybe we know now why Matt Cutts doesn't use one? :p

After assuring one member that the robots.txt was properly formatted, critter gets some further support is his assertion. I will be examining some log files over the next week to see for any such instances, and I'm sure we would all love to see more evidence of this in the thread.

Read about it and post your thoughts at Search Engine Watch Forums.

See Rand’s post at SEOmoz Blog.

 

Popular Categories

The Pulse of the search community

Google Search Volatility

More Details

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google Adopts New AI Training Control

Oct 30, 2026 - 12:50 pm
Bing SEO

Microsoft Bing New AI New Controls With AI Training From AI Search Grounding

Oct 30, 2026 - 12:15 pm
ChatGPT Ads

ChatGPT Gets Wider Ads, Multi-Image Ads Expands, Expansion & More

Oct 30, 2026 - 11:17 am
Search Forum Recap

Daily Search Forum Recap: October 30, 2026

Oct 30, 2026 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Volatility X2, Spam Updates, AI Overviews Everywhere, Blending Ads & Organic, Google & Microsoft Earnings

Oct 30, 2026 - 8:01 am
Google Search Engine Optimization

Google AI Overviews & AI Mode 0.9% Click-Through Rate, Opt Out Study Says

Oct 30, 2026 - 7:51 am
Previous Story: WebmasterWorld Conference Pass Giveaway