In a live Google hangout with John Mueller this morning on Google+, John said that Google doesn't set up test sites for the most part to test and experiment with the search results. Instead, they test on a small percentage of live users, like 1% or so, so they can collect the data in a real world experiment.
Truth is, most of you know this is how Google works.
The question came in at the 10:15 minute mark:
Googlers regularly talk about experimenting with algorithm changes on live. Is there a test set of sites used, who does the testing (real serps observed, testers, bots?) and what is the impact on sites involved in testing?
John said:
In general, we don’t have a limited set of test sites for something like this. It is because the web is evolving all the time. So we try to do things like 1% launches to see what happens when we redirect a small portion of the traffic to a specific change. We try to run experiments across pretty much all users to see how the general reaction is to these changes.So it is not really limited to a number of sites because I think that would artificially skew a little bit if there was this kind of fixed set of sites that everything is judged against.
Again, this isn't really new but maybe, just maybe what we saw earlier this week with a potential Penguin launch was just a small limited test.
You can watch the video here:
Forum discussion at Google+.
