The Search Engine Relevancy Challenge Live: RustySearch

May 3, 2005 • 8:35 am | comments (5) by twitter Google+ | Filed Under Search Engine & SEO Theory

A few weeks ago, I proposed the idea of The Search Engine Relevancy Challenge: Pepsi vs. Coke. In this challenge, I proposed building a "white labeled" search engine, that randomly selected results from one of the top four engines and asked you to rate the search engine results, individually, from one to five. Well, we have built the white label search engine.

Officially, this is being released tomorrow, but I wanted you guys to get to it first. There is more information at the official, The Search Engine Relevancy Challenge. I named the white labeled engine RustySearch and it works as follows:

1. Go to RustySearch (will open in new window) 2. Enter a search phrase and click search 3. Click on the title, the URL or the "rate this result" link. The site will open up in a framed window, so do not open the result in a new window. 4. At the top, there is a frame that says, "Rate the relevancy of this Web page for 'search phrase here'". Rate the result from one to five, one meaning a very poor relevancy and five meaning an excellent relevancy. 5. After you click a radio option, it should read, "Thanks for your input! Your vote (and many others) will decide who, really, is the search engine king!" 6. Then click on the button to take you back to the results 7. Continue on by rating the next result on that page, or search on a different result. 8. Repeat...

It is close to impossible to make everyone happy with a study on relevancy. Please try not to search on terms you are looking to rank well for. Search on terms when you have a information search need, outside of rank checking. This way, you can really test the relevancy as an end user and not an SEO.

I am off to Toronto SES, speak to you all tomorrow!

Previous story: Tiger Here



05/03/2005 03:27 pm

Was thinking it might be easier to rate the page on a scale of 1-10 instead of having to vote 10 times on a page...could be as simple as how many of the top 10 are good pages? maybe even 2 many are good, and how many are bad (rest would be "so so" pages)...just an idea.

Barry Schwartz

05/03/2005 03:32 pm

Forcing the user to really see if the results and really spend time thinking if it is relevant. You don't have to rate all 10, just rate the ones you want to rate.

Stoney deGeyter

05/03/2005 06:29 pm

The fact that results are numbered and in vertical alignment might tilt the results. There is a basic assumption (intended or not) that the higher up the more relevant a result is. I would recomend a grid layout, say three rows of three.


05/04/2005 05:31 pm

Hey Barry, the rustysearch logo looks a lot like the one at Is that a stock graphic? Just curious. Nice viral campaign. Am looking forward to the results. Lee

Barry Schwartz

05/04/2005 06:17 pm

Stock images, so you know. Nothing gets more "White Label" then stock art.

blog comments powered by Disqus