(5)
A Cre8asite Forums thread links to a blog post named GoogleBot Requested a CSS File. This is not the first time I heard threads where people suspect GoogleBot is crawling their CSS files. But this one has the most discussion I have seen so far.
Some people in the thread think a manual review sparked the crawl. But I am not too sure about that.
The request in the log file looks like this:
66.249.72.52 - - [24/Oct/2006:17:17:35 -0500] “GET /global/x.css HTTP/1.1″ 200 8382 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
So it looks like Google crawled his CSS. But he never discusses if he disallowed the directory in his robots.txt file.
Anyway, for past articles on Google and CSS see:
- Does Google Read Your CSS Files? - November 11, 2005
- Hidden Divs Not Indexed by Google - December 15, 2005
- CSS, AJAX, Web 2.0 & Search Engines - SES Session - December 6, 2006
- Using CSS To Hide Text: Search Engine Responses - December 18, 2006
Forum discussion at Cre8asite Forums.
Comments:
Pierre Far
01/05/2007 01:03 pm
Pierre here from eKstreme.com. Re the robots.txt block: Nothing in the robots file blocks the CSS directory. As I mentioned in my blog post, I do this check regularly, so I made sure I didn't block access :)
SearcH EngineS WeB
01/05/2007 01:37 pm
http://www.mattcutts.com/blog/communication-in-other-languages/ This had been hinted/ implied last month in this high profile Blog Post Either that, or the spam team is MANUALLY seaching through CSS files if they detect hidden text.
Peter van der Graaf
01/05/2007 02:16 pm
A robots.txt disalow of your stylesheets is disregarded by most search engines. Not that I'm saying you should cloak, but cloaking your stylesheet is something that is hard to detect by both competitors and search engines. <a href="http://www.vdgraaf.info/when-should-i-use-cloaking.html#javascript">http://www.vdgraaf.info/when-should-i-use-cloaking.html#javascript</a> Nevertheless, prevent using any CSS spam tactics. There are many better ways to fool people and search engines.
Michael Martinez
01/05/2007 04:53 pm
They have been fetching CSS and Javascript files for years. I and other people have been confirming as much for years.
Pierre Far
01/05/2007 07:57 pm
I just updated the blog post: external Javscript files were also downloaded. All of the JS downloades happened in a two-month window between September and November last year. Was all this some sort of experiment? Pierre