As we know, algorithms are very powerful in influencing the search results. However, we also know that Google has human evaluators -- thousands, in fact. In a recent blog post on the Official Google Blog, one of Google's engineering directors shed some light into the role of human evaluators. The relevant piece of information is here:
Google makes use of evaluators in many countries and languages. These evaluators are carefully trained and are asked to evaluate the quality of search results in several different ways. We sometimes show evaluators whole result sets by themselves or "side by side" with alternatives; in other cases, we show evaluators a single result at a time for a query and ask them to rate its quality along various dimensions.
Tedster at WebmasterWorld is pretty interested in these results; the fact that human evaluators actually look at results side by side to help improve the quality of its search results is pretty reassuring.
Forum discussion continues at WebmasterWorld.
