At SMX West a few weeks ago, during Paul Haahr's awesome presentation and Q&A, someone asked if Google can actually dig into the search quality rater's results. I believe Paul explained that when a quality rater result set shows a certain search result as being quality or not but the engineers don't necessarily believe it, they can dig deeper and see the details.
For example, let's say Google is testing a new ranking signal and overall it shows that the new ranking signal is working well. Google can then dig into the experiments run, see the queries they entered, check the results the raters rated and verify it was done correctly.
So they can not just see the macro level results but also dig deeper into the micro actions to see how the overall analysis came about.
Sometimes, Paul said, it makes you reconsider launching the new ranking signal because the quality raters were not exactly right.
Here are some tweets covering that part:
Google has the ability to drill down into the individual query level when doing experiment analysis says @haahr #smx
— AJ Kohn (@ajkohn) March 3, 2016
Google sees which experiments are winners or losers. But there are some caveats. #smx @haahr
— Jennifer Slegg (@jenstar) March 3, 2016
Forum discussion at Twitter.
