Google Has Whitelists & Exception Lists

Mar 14, 2011 • 9:16 am | comments (10) by twitter Google+ | Filed Under Google Search Engine Optimization
 

Google Whitelists/Exception ListsIn January we asked if Google has a whitelist for their search engine. Then a couple weeks ago, Google said no, they don't have whitelists - generally. Yes, a vague answer.

So at The Spam Police panel at SMX West, I submitted a question for Google's Matt Cutts, asking him out right, does Google have whitelists.

Matt went on to answer in a long way, but basically said, yes, Google does have whitelists for what they call, "exception lists." But it is important to note that the exception lists are on a per-algorithm basis. Meaning there is not a single specific global whitelist where a site will never get hurt in the Google results - some people call it the Wikipedia whitelist. Google said that most of the algorithms don't even have whitelists, but some do and have to because not all search algorithms are a 100% perfect.

Danny and I wrote in more detail about this at Search Engine Land and then Google provided a statement:

Our goal is to provide people with the most relevant answers as quickly as possible, and we do that primarily with computer algorithms. In our experience, algorithms generate much better results than humans ranking websites page by page. And given the hundreds of millions of queries we get every day, it wouldn't be feasible to handle them manually anyway. That said, we do sometimes take manual action to deal with problems like malware and copyright infringement. Like other search engines (including Microsoft's Bing), we also use exception lists when specific algorithms inadvertently impact websites, and when we believe an exception list will significantly improve search quality. We don't keep a master list protecting certain sites from all changes to our algorithms.

The most common manual exceptions we make are for sites that get caught by SafeSearch-a tool that gives people a way to filter adult content from their results. For example, "essex.edu" was incorrectly flagged by our SafeSearch algorithms because it contains the word "sex." On the rare occasions we make manual exceptions, we go to great lengths to apply our quality standards and guidelines fairly to all websites.

Of course, we would much prefer not to make any manual changes and not to maintain any exception lists. But search is still in its infancy, and our algorithms can't answer all questions.

Yes, Microsoft said they also use exception lists. But those lists are always revised and cleaned off. I.e. when they push out an update to an algorithm, they try to make sure the sites in the exception list can be removed from the list, thus perfecting that algorithm.

To me, this is a big deal for the SEO space and I am not sure why it didn't get more play yet.

I might be able to publish a video recording of the session where Google and Bing admitted to having this exception list, so check back later today.

Forum discussion at Google Webmaster Help.

Update: I posted the full transcript and audio at Search Engine Land - enjoy!

Previous story: When To Use 404, NoIndex & 301 Redirects In Farmer/Panda Fix
 

Comments:

Nick Stamoulis

03/14/2011 02:39 pm

No algorithm is perfect. There are plenty of unhappy site owners who feel like they were inappropriately targeted by the Panda update. I assume some sites get caught by the algorithm more frequently than others (like essex.edu getting flagged for having 'sex' in the address) and those are the ones Google has on their "white list" of quality sites that accidentally get negatively affected.

Michael Martinez

03/14/2011 04:53 pm

I think both Microsoft and Google explained the process adequately. I was sitting in the audience at that panel and as a programmer myself I can well understand the need to incorporate the occasional exception list when you need to get an otherwise good algorithm out the door. I just wish they had been more upfront about this much, much sooner.

Barry Schwartz

03/14/2011 04:54 pm

I believe I asked the question because of you.

Randy Pickard

03/14/2011 05:36 pm

This is a bit off topic, but offers another example of Google making a manual change. A search on the brand name, iNest, would return "were you looking for incest". Not exactly the association we were looking for. It required multiple requests before Google finally reacted and discontinued returning the opportunity to search for incest to visitors searching for iNest, but they did respond to our requests.

Ryan Bradley

03/14/2011 05:37 pm

I think the reason why most people are upset with lists like these is that they seem unfair. Could a company potentially buy their way on this list? I personally don't think they are selling spots on their list but it raises a lot of "what ifs" since the list is secret and people know it exists.

Ben Pfeiffer

03/14/2011 06:04 pm

It's about time they admitted to the lists.

@JoeHobot

03/14/2011 06:48 pm

Pay them, and you will be on that white list too :) It's business man. You think really huffpost, ehow and about will just let google dominate their search quality for specific keywords that they have been no.1 ranking for on google? Its 21st century, you charge everything, and nothing is free.

online shopping uk

03/15/2011 07:52 am

Yes You are Right.. No Algorithm is perfect, There is always roam for improvement.. But their are many unhappy siteowners are still there, even reputed sites like ezine articles also effected with this update..

Felipe

03/16/2011 09:15 am

I don't see why people have a problem with this. It's their search engine and if they feel the need to have whitelists, whether paid or just for the sake of protecting sites that are wrongfully excluded from restults, then so be it....What else can we say?

Steve

03/16/2011 07:13 pm

The perspective of those who build the systems always differs so greatly from those who just exist within the system. Makes me wish more SEOs would have a chance to walk in the programmers' shoes. The insight it would give into systems (such as search engines) could be tremendous.

blog comments powered by Disqus