Early Results from RustySearch

May 6, 2005 • 12:25 pm | comments (10) by twitter Google+ | Filed Under Search Engine & SEO Theory

So which search engine is most relevant after 3,244 rated search results? Here are some charts I put together for a quick preview of what is going on with The Search Engine Relevancy Challenge. Have your friends, family and the person working with you use it, so we can get a more representative sample. We are testing relevancy, not if my site ranks number one in this engine or not.

This pie chart shows the "most relevant" search engine by averaging the scores of all the ratings. The search engine listed at the top, is the most relevant engine. The number to the right of the name in the legend is the average rating for that seach engine. We then plotted that data in the pie chart.


This link graph groups search engines by rating. The reason we plotted it on a graph like this is to show you that there is this U shaped curve that is consistent between all search engines when rated. In our opinion, it means that most people either feel the results are relevant or not relevant. Very few people feel that a search engine can be "somewhat" relevant.


Finally, here is a raw summary count of data that we placed on a simple chart view for you. This data is real time and will continue to update as people rate. The averages from top-down are the average rating count by search engine. The averages from left-right are the average rating count by rating group. The value at the far bottom-right corner is the total rated search results obtain at this point in time at RustySearch.

Raw Summary Data
Search Engine12345Average
Ask Jeeves248737761329157.60
MSN Search291665676314160.60

Previous story: Catching Up After SES Toronto 2005



05/06/2005 05:18 pm

Very interesting research Rusty. Took me quite awile to go through the first page of the results and read everything since I was looking for very specific legal information. I'm surprized that these SEs are so close to each other. Looks like Google stands out pretty strong in rating group 4. I think if we had more specific evaluation instructions, than this would be very accurate analyses. I'm not fully satisfied with somewhat relevant results because they don't answer my request in full, like when I was looking for legal information. So, if I need to make a correct decision and group #5 means correct answer - everything else just dont do it for me. So I think relevancy should be described and valued according to those descriptions/instructions since very often we can get irrelevan/somewhat relevant result that mention requested topic on the page but not explain it. I like the research - thank you!


05/06/2005 05:59 pm

The problem with this type of test is that you are measuring user's perception of relevancy, not what a search engine scores as relevancy. This mismatch has nothing to do with sample size. See Nacho's post at this site. Each SE has its own scoring functions, so it is like comparing apples with oranges. I wish you luck with the experiment.


05/06/2005 06:16 pm

is there a link to Nacho's post? Well, I don't think there is a way to mesure search engine's score of relevancy unless you are a search engine. Search Engines for people, to retrieve relevant information on their requests, so it should be based on user's perception of relevancy and of course users have to know how to make correct queries. Of course each engine has their own scoring functions and I'm sure they are different that's why this research based on human jugement.. and that's why I though it would be much better to come up with more specific relevancy scale based on user's perception of relevancy again. It doesn't compare their own relevacy scores between each other, so why would it be like apples to oranges?


05/06/2005 06:30 pm

Found the post.

Chris B

05/06/2005 06:34 pm

Very cool! I did about 5 searches, and found that "search engine optimization" has the most 5's, in my opinion. I guess that is good. I am surprised at the dramatically lower number of 2, 3, and 4's, since I used a fair amount of them myself. Some sites I rated as 1's or 2's were done so because they were the second results from the same site, which has always bothered me. Any site that required a login for more information received a 1 from me. I found a lack of relevant results for the term "diamond pricing guide," with only about three fitting the bill, and the rest simply some dealer's page. It is my opinion that terms including "guide," "review," or "comparisons" should lead to objective sites. Interesting that MSN led the pack in 1's and came in last in 5's...I guess they do have some work to do...thanks again Barry!

Mikkel deMib Svendsen

05/06/2005 07:08 pm

> The problem with this type of test is that you are measuring user's perception of relevancy, not what a search engine scores as relevancy. Orian, I think you totally misunderstand the purpose of this test - the test is exactly about user perception NOT about how engines score results. I don't (as a user) care one bit about how they score only about what I get - the relevancy from my perspective. This test is as perfect as such gets!

Barry Schwartz

05/06/2005 07:36 pm

I agree with Mikkel, of course. :) My main issue right now is that most "SEOs" are using the RustySearch test. I am doing my best to get the word out to non SEOs. This way I can get a better sample.


05/06/2005 08:11 pm

Mikkel, I agree with you. I think my misunderstanding is due to the way its is worded. If is about users perception then it should be referenced in that way. If the findings are triggered by what the users perceive as relevant then is not about search engine relevancy, which are scoring values. I do believe the test has some valid merits. Orion


05/10/2005 11:41 am

Cool research. I'll include some of this stuff in my blog. http://fortuneaffiliate.blogspot.com Thanks, John


05/11/2005 03:37 am

Each search engine will claim it serves the most relevant results according its own criteria of relevance. What we need is cross comparison by users like one. Great work.

blog comments powered by Disqus