Google: Do Not Do Extended A/B Testing With The Same A/B Test

We've covered how Google handles A/B testing and that they are cool with it. But someone asked in a Google Hangout on Google+ how long is too long for an A/B test. John Mueller from Google wouldn't give a specific time frame but he said don't let it stick for too long.

This question came up at the 15:04 mark into the video:

I’m serving A/B tests to a big proportion of traffic using Adobe software. If these tests are served to a selection of UA but not Googlebot, is this considered cloaking and could it risk a penalty? What is the recommended duration of time for a A/B test?

John responded:

So from our point of view, you should treat GoogleBot like you would any other user. So if you are doing A/B testing then GoogleBot probably needs to fall into one of those A or B test, how many variations you are testing slots as well.

The other thing to keep in mind is you need to kind of treat GoogleBot like other users in that location. So if you treat users in a different way, if you do an A/B test specifically for users in the US and GoogleBot crawls from the US for your site then you need kind of need to include GoogleBot in that A/B test route. In other way, if you are doing an A/B test for users in France and GoogleBot is crawling from the US and everyone from the US sees the same version then that is nothing where you’d need to do anything special.

With regards for the duration, we just recommend to kind of keep it reasonable. In the sense that you shouldn’t be doing an A/B test for an extended period of time where you can say well, it is not an A/B test anymore, it is essentially the stable state for the site. So that is something where we don’t have any fixed time but kind of need to look at it on your site.

Some sites do regular A/B testing, that they go A/B and that they test it for a month and then they test other variations and next variations and they continually test and that is fine too. It is just we don’t want a situation where you do different content from what GoogleBot uses.

So he said don't do it for an "extended period of time" but you can continue to A/B test different variations all the time, like Google does.

It is more about a cloaking issue that you should avoid.

Here is the video embed:

Forum discussion at a href="https://plus.google.com/u/0/events/c2vhkbnl9dnso7jacf2ncv7o098">Google+.