The Search Engine Relevancy Challenge: Pepsi vs. Coke

Apr 13, 2005 • 4:49 pm | comments (10) by twitter Google+ | Filed Under Search Engine & SEO Theory

As a follow up to my entry named Google Relevancy Poor Based on Generalization, I have started a new thread at Search Engine Watch Forums named Coke vs. Pepsi Challenge for Search Engines.

I understand that relevancy is extremely subjective, however, search engines all want to achieve the most optimal level of relevancy possible. The engines all want their results to be the most relevant for each individual searcher, no matter who it is. Based on that, I figured it would be cool to set up a Coke versus Pepsi challenge, but for the search engines. Basically, the concept would be to create a "white labeled" search engine page, allow users to query the search engine and return results from one of the big four (Google, Yahoo!, MSN and Ask Jeeves). Then the user would mark which results are relevant on some sort of scale. Collect this data over time and then show the results.

Currently, many search engine users accept that Google or Yahoo! or the engine of their choice is relevant. So if I am a Google user, I will compare those results to Yahoo! and say, wow Yahoo!'s results are far less relevant to me then Google. Relevancy, as mentioned above, is extremely subjective. And with subjective concepts, comes bias. Search engines spend a lot on branding, to influence user bias. If we take branding out of the equation, I strongly wonder what the user would consider relevant and what the user will consider irrelevant. Even more so, I wonder which of the four engines would win the contest.


I am strongly considering building a white labeled search engine to do this. Since Google and Yahoo! both have APIs and MSN has an RSS feed, I can make it work. In addition, I think Ask Jeeves would be happy to provide some sort of feed. If you are interested in building out this white labeled search engine to collect subjective relevancy data, feel free to let me know. This can be fun!

Previous story: Domain Names & Hyphens, Lots of Hyphens



04/13/2005 10:51 pm

An interesting theory.

Razvan Atonescu

04/14/2005 07:25 am

Excellent ideea. A few months ago I had a presentation where the core ideeas were: 1. No search engine is relevant on all domains and therefore a relevancy test should be performed on all SE before conducting a research 2. For the fans of a particular SE. If they are presented with sets of results from all SE without a logo they could find relevant the results from other SE other than that that is used on a regular basis 3. Google is smthg like Xerox and has become associated with the process of search but that doesn't make it the most relevant

Razvan Atonescu

04/14/2005 07:32 am

regarding this statement: "Even more so, I wonder which of the four engines would win the contest" I don't think that it will be an overall winner but it should be made a ladder based on custom fields of interest. For example DMOZ main categories

Barry Schwartz

04/14/2005 12:10 pm

I would love to see the presentation. Care to share?


04/15/2005 05:03 am

Hi, Try this one Note: does not work right now, but can be easily fixed. When you click on one of the stars, it will mark same URL in the other columns. This actually remembers your favorites and lights up the stars when you come back (if you are logged on). This was build about 6 months ago and was not touched since then. Could be updated to match what you are looking for. What do you think?

Barry Schwartz

04/15/2005 01:52 pm

I think this can work well. Let me show a few others...


04/15/2005 11:52 pm

If you want to update this and make changes, email me.

Dan Thies

05/12/2005 07:14 pm

Comments and a suggestion over at my blog, Barry...

Peter Lucas

11/16/2005 02:34 am

Great idea, but is it really about relevancy? If people are searching for things they value or see value in then should we not be meauring value rather than relevancy. I mean I can search for "Software Development" and even though I might get relevant listings based on my original query and thoughts, reading an article of a docuemnt listed 7 on the search ranking may broaden my knowledge and mean this article while ranked less relevant than position 1, is actually more relevant to me. So, maybe relevancy is one of several elements in the makeup of 'value', which would mean a whole lot of things......but most importantly it means relevancy should be less of a focus and value should be more of a focus. Just a thought.....


10/23/2007 05:45 am

I am participat in contest. please participate on contest.. thank you....

blog comments powered by Disqus