Barry, is it wise to block a robot from a CSS file, or is that even what you meant?

Logically this makes little sense, but I'll make sure I'm using external style sheets I guess ...

Yea, that is what I meant. Clock the external CSS files.

Interesting -- but is that really the purpose of robots.txt?

I thought it was to tell them what not to _index_. Not necessarily what not to look at it. And it's an "honor system." I doubt they're really obligated to listen to you, are they? It would never have occured to me to block a CSS file from a spider, and quite honestly, if I were writing an algorithm, that would be something I would be hestitant to honor -- it's kinda suspicious ...

I mean, I could create lots of spam sites with that technique. Maybe I should put on a black hat after all ..

Jaimie, Google will not have spiders touch files that have been excluded via the robots.txt file. They are very strict about it. The only spiders that don't care are rogue spiders.

Is it fair to say, then, that this technique could be (very) successfully used to cloak?

That's my only problem with this line of reasoning -- why bother with Fantomaster when you can just block a css file that makes your viagra spam invisible to everyone except Google ...

No its not fair to say that.

I think that people forget that many times when an SE suspects funny stuff, they can easily have a real person go look at the file easily. Blocking the robots file would only stop automatic filtering of the css hidden properties.

I know it's possible for this to happen, but in practice, it doesn't. There are too many spam sites in too many prominent areas for me to believe that Google directs a lot of effort into this. Of course I may be wrong.

But Barry, to be quite honest, if I can scrape a few results off of some engine, scrape those sites for content, display:none it in a stylesheet that's blocked by an excluded file in robots.txt, then use some 2nd-tier PPC (searchfeed?) feed to monetize the spam, wouldn't this be a clear and flagrant reason for Google _NOT_ to honor my exclusion?

They can manually review your site and boot you, like William said above.

Just wanted to point out on ducor.com that the site is blocking cookies, and Yahoo is also showing some pages with the error page cached. It seems the site is blocking cookies and they don't have an exception for spiders. Its part of the problem at least.

The same can be said about Fantomaster, no?

In fact, I'd say this method, though in some ways primitive, might be HARDER to detect than Fantomaster-style IP-origin cloaking.

Just a thought ...

I'm totally not a black hat spammer, and rather ignorant of the "dark side," but TBH, I don't see cloaking having a major edge over clever CSS spam.

Does Fantomaster know the IPs of everyone at Google -- not just the spiders? Meaning, their "QA" team? I doubt it ...

Let's leave Fantomaster out of this.

I would go further and ban G ip from accessing the css file through .htaccess

I have been able to look into this a bit more since posting at SEW and have found that in all the text caches I have looked at of pages that use this type of menu system are missing the contents of the hidden divs used in the menu system.

But similar menu systems where the hidden divs are declared in the CSS file instead of on the page are intact.

Can it be long until Google dumps all these too?

IMO this is a boo boo - thowing out the baby with the bathwater type of thing.

I wouldn't agree with that. But I'm whitehat. I guess it's one extra stopgap measure, though.

Though Barry seems to indicate that G won't touch it anyway. And if they manually audit your site, I'm sure they won't use Google IPs as I've said above.

J.

This is not about spamming. It is about those who use the divs for non-spam reasons and are totally unaware of this. So if a manual audit would occur, the manual review will show a clean site that got hit for no reason.

I don't buy it. vBulletin legitimately uses hidden divs for it's DHTML drop-down menus and every vBulletin forum I look at (including mine) is not banned from Google.

Shawn, I was thinking the same thing, about 4 hours ago. But all I do is cover the buzz, not come up with it. A very valid point, however, I also thought that Google has something in there that is special for vb forums, since its so widely used...

Shawn,

It's not causing banning. People aren't saying it gets you banned, they're saying it just doesn't get indexed. This means if you have some links in DIVs and they're visibility:hidden, Google is ignoring it. This is akin to the fad in the 90s of using javascript menus that didn't get spidered.

And I'm not a blackhat. I'm a whitehat. Barry, I'd argue that in order to be a competent SEO, you'd have to also analyze this as a blackhat. If you don't, you aren't fully understanding G's motives.

So it IS about spamming IMHO, because spammers are what caused this aggressive tactic.

J.

You can nearly always blame the spammers... anyway I wouldn't necessarily recommend invivisble divs for menus because it's somewhat predicted that they will not be seen or indexed by bots, as it probably happened now.

But how many thousands of sites out there that use image replacement for legitimate reasons (on-page content == image content) would actually receive a manual audit? I seriously doubt that even 5% would; their SERPs would just tank for no reason.

Jaimie - Whatever the case may be, I don't buy it. :)

My forum uses openly hidden div, and there are 576,000 pages in the Google index, the content of those divs are seen in Google's cache (and searchable with a web search), and the links within them are also seen with the link: command.

It *may* be one of the million factors Google uses. Probably individually it doesn't trigger a penalty, but if you combine it with other stuff maybe it does. {shrug}

Shawn,

Barry suggested something I'd hesitate to believe ... that they hacked in faux whitelisting for common forum posting software. TBH, it looks like that to me.

Thing is, from a programmign point of view .. a purist designing an algorithm designed to function generally would be tainted by hacks like that.

That said, I used to openly go style='display:none.' I stopped that. And I'm putting my css in a folder and denying bots access. Cloaking a 404 for the file is a bit overboard, because as Barry said, SEs honor robots.txt rather well ...

Jaimie.

This is about spamming??? (probably is.) This seems a tremendous overkill which will affect many clean sites. Seriously, if a spammer can't put his css in an external file... is he/she really capable of creating spam that's threatening to Google?

Ok, so great move G. Spammers will now move to external css and the ratio of clean to dirty sites hit by this will keep (un)improving!

Its not just about menus, right? Hiding and displaying content via toggles makes for a more efficient, user-friendly UI. It is akin to a tabbed browser, or multiple application windows on a desktop.

I trust Google spiders will accomodate such UI, though it may mean the spider has to do a logical analysis of the css and javascript. If not, they set the web back.

EP

Hmm no, I can confirm that hidden divs are indexed by Google and they even show up in search results. See for yourself
http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=6VQ&q=http%3A%2F%2Fwww.thinkwiki.org%2Fwiki%2FCategory%3AModels&btnG=Search
First result, notice description, go to the page, view source - the description is in a hidden div.

"So what can you do now? Move that CSS off the page, to an external style sheet. Then maybe, block the robots.txt from those files."

that entire last line shows how little the author knew/knows about CSS/SEO/Google/Search Engines. The fact they are blogging about something they know little about is funny though