Very interesting research Rusty. Took me quite awile to go through the first page of the results and read everything since I was looking for very specific legal information. I'm surprized that these SEs are so close to each other. Looks like Google stands out pretty strong in rating group 4. I think if we had more specific evaluation instructions, than this would be very accurate analyses. I'm not fully satisfied with somewhat relevant results because they don't answer my request in full, like when I was looking for legal information. So, if I need to make a correct decision and group #5 means correct answer - everything else just dont do it for me. So I think relevancy should be described and valued according to those descriptions/instructions since very often we can get irrelevan/somewhat relevant result that mention requested topic on the page but not explain it. I like the research - thank you!

The problem with this type of test is that you are measuring user's perception of relevancy, not what a search engine scores as relevancy. This mismatch has nothing to do with sample size. See Nacho's post at this site.

Each SE has its own scoring functions, so it is like comparing apples with oranges. I wish you luck with the experiment.

is there a link to Nacho's post?

Well, I don't think there is a way to mesure search engine's score of relevancy unless you are a search engine. Search Engines for people, to retrieve relevant information on their requests, so it should be based on user's perception of relevancy and of course users have to know how to make correct queries. Of course each engine has their own scoring functions and I'm sure they are different that's why this research based on human jugement.. and that's why I though it would be much better to come up with more specific relevancy scale based on user's perception of relevancy again. It doesn't compare their own relevacy scores between each other, so why would it be like apples to oranges?

Found the post.

Very cool! I did about 5 searches, and found that "search engine optimization" has the most 5's, in my opinion. I guess that is good. I am surprised at the dramatically lower number of 2, 3, and 4's, since I used a fair amount of them myself. Some sites I rated as 1's or 2's were done so because they were the second results from the same site, which has always bothered me. Any site that required a login for more information received a 1 from me. I found a lack of relevant results for the term "diamond pricing guide," with only about three fitting the bill, and the rest simply some dealer's page. It is my opinion that terms including "guide," "review," or "comparisons" should lead to objective sites. Interesting that MSN led the pack in 1's and came in last in 5's...I guess they do have some work to do...thanks again Barry!

> The problem with this type of test is that you are measuring user's perception of relevancy, not what a search engine scores as relevancy.

Orian, I think you totally misunderstand the purpose of this test - the test is exactly about user perception NOT about how engines score results. I don't (as a user) care one bit about how they score only about what I get - the relevancy from my perspective.

This test is as perfect as such gets!

I agree with Mikkel, of course. :)

My main issue right now is that most "SEOs" are using the RustySearch test. I am doing my best to get the word out to non SEOs. This way I can get a better sample.

Mikkel, I agree with you. I think my misunderstanding is due to the way its is worded. If is about users perception then it should be referenced in that way. If the findings are triggered by what the users perceive as relevant then is not about search engine relevancy, which are scoring values.

I do believe the test has some valid merits.

Orion

Cool research. I'll include some of this stuff in my blog. http://fortuneaffiliate.blogspot.com

Thanks,

John

Each search engine will claim it serves the most relevant results according its own criteria of relevance. What we need is cross comparison by users like one. Great work.