More Tidbits on Google's Duplicate Content Filter | Main | Best Search Marketing Blog Award From Search Engine Journal

January 5, 2007

Is Google Sending GoogleBot CSS Hunting: Google Crawling CSS Files?

A Cre8asite Forums thread links to a blog post named GoogleBot Requested a CSS File. This is not the first time I heard threads where people suspect GoogleBot is crawling their CSS files. But this one has the most discussion I have seen so far.

Some people in the thread think a manual review sparked the crawl. But I am not too sure about that.

The request in the log file looks like this:

66.249.72.52 - - [24/Oct/2006:17:17:35 -0500] “GET /global/x.css HTTP/1.1″ 200 8382 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

So it looks like Google crawled his CSS. But he never discusses if he disallowed the directory in his robots.txt file.

Anyway, for past articles on Google and CSS see:

Forum discussion at Cre8asite Forums.



Like The Story? Vote For It On Yahoo Buzz! Or On Sphinn!

posted rustybrick in Google Optimization at January 5, 2007 7:51 AM Comments (5)

• Google Co-op Cartoon Barry

Comments

Pierre here from eKstreme.com.

Re the robots.txt block: Nothing in the robots file blocks the CSS directory. As I mentioned in my blog post, I do this check regularly, so I made sure I didn't block access :)

 

Posted by Pierre Far at January 5, 2007 08:03

http://www.mattcutts.com/blog/communication-in-other-languages/

This had been hinted/ implied last month in this high profile Blog Post

Either that, or the spam team is MANUALLY seaching through CSS files if they detect hidden text.

 

Posted by SearcH EngineS WeB at January 5, 2007 08:37

A robots.txt disalow of your stylesheets is disregarded by most search engines.
Not that I'm saying you should cloak, but cloaking your stylesheet is something that is hard to detect by both competitors and search engines. http://www.vdgraaf.info/when-should-i-use-cloaking.html#javascript

Nevertheless, prevent using any CSS spam tactics. There are many better ways to fool people and search engines.

 

Posted by Peter van der Graaf at January 5, 2007 09:16

They have been fetching CSS and Javascript files for years. I and other people have been confirming as much for years.

 

Posted by Michael Martinez at January 5, 2007 11:53

I just updated the blog post: external Javscript files were also downloaded. All of the JS downloades happened in a two-month window between September and November last year.

Was all this some sort of experiment?

Pierre

 

Posted by Pierre Far at January 5, 2007 14:57

Post a comment (Note: Can Take 120 Seconds For Your Comment To Show Up)

Do you want us to save your personal Information?

Premium Sponsors + advertise

To subscribe to the Search Engine Roundtable, click here