Google's John Mueller noted that if your A/B tests are "significantly different" from one and the other, then that "could be visible in search results too." He also said, "Depending on your setup, what might happen is that one or the other version is used for indexing."
He posted this on Bluesky when he was asked the following question:
As Google's A/B testing guide suggests to avoid running same A/B test for long durations, I was wondering how does Google handle long term holdouts (eg. 10% for 6-12 months), especially for a large scale marketplace with 10s of millions of crawls to similar amount of pages.
John Mueller replied:
Depending on your setup, what might happen is that one or the other version is used for indexing. If they're close enough, probably that doesn't matter. If they're significantly different, that could be visible in search results too.
The follow up question was:
What if it's fully different like a redesigned page, and since Googlebot is getting alternative versions with each crawl (sometimes in a day). Can that rapid change in core HTML structure cause issues with indexing and lead to Google potentially dropping the pages from index?
John responsed:
We'd take the content into account the way that we crawl it for indexing. There's no (as far as I know) "penalty" or "demotion" for having varying content (lots of sites have that), but it can make it harder for you to debug & monitor if the content constantly changes.
Now, Google has an official A/B Testing best practices guide that does say "Small changes, such as the size, color, or placement of a button or image, or the text of your "call to action" ("Add to cart" vs. "Buy now!"), can have a surprising impact on users' interactions with your page, but often have little or no impact on that page's search result snippet or ranking."
Google set up that document in 2022 after giving advice on the topic for years on A/B testing, in blog posts and on videos. Google felt it needed an A/B testing guide and made one.
That being said, if you do plan doing A/B testing and you care about Google Search and SEO, check out Google's official guide.
Forum discussion at Bluesky.
