Is Google Sending GoogleBot CSS Hunting: Google Crawling CSS Files?

Jan 5, 2007 • 7:51 am | comments (5) by twitter Google+ | Filed Under Google Search Engine Optimization

A Cre8asite Forums thread links to a blog post named GoogleBot Requested a CSS File. This is not the first time I heard threads where people suspect GoogleBot is crawling their CSS files. But this one has the most discussion I have seen so far.

Some people in the thread think a manual review sparked the crawl. But I am not too sure about that.

The request in the log file looks like this: - - [24/Oct/2006:17:17:35 -0500] “GET /global/x.css HTTP/1.1″ 200 8382 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +"

So it looks like Google crawled his CSS. But he never discusses if he disallowed the directory in his robots.txt file.

Anyway, for past articles on Google and CSS see:

Forum discussion at Cre8asite Forums.

Previous story: More Tidbits on Google's Duplicate Content Filter


Pierre Far

01/05/2007 01:03 pm

Pierre here from Re the robots.txt block: Nothing in the robots file blocks the CSS directory. As I mentioned in my blog post, I do this check regularly, so I made sure I didn't block access :)

SearcH EngineS WeB

01/05/2007 01:37 pm This had been hinted/ implied last month in this high profile Blog Post Either that, or the spam team is MANUALLY seaching through CSS files if they detect hidden text.

Peter van der Graaf

01/05/2007 02:16 pm

A robots.txt disalow of your stylesheets is disregarded by most search engines. Not that I'm saying you should cloak, but cloaking your stylesheet is something that is hard to detect by both competitors and search engines. <a href=""></a> Nevertheless, prevent using any CSS spam tactics. There are many better ways to fool people and search engines.

Michael Martinez

01/05/2007 04:53 pm

They have been fetching CSS and Javascript files for years. I and other people have been confirming as much for years.

Pierre Far

01/05/2007 07:57 pm

I just updated the blog post: external Javscript files were also downloaded. All of the JS downloades happened in a two-month window between September and November last year. Was all this some sort of experiment? Pierre

blog comments powered by Disqus