I'm meddling in Bill Slawski's territory here, in that he's better known for presenting and analyzing papers on search engine technology. However, I caught this one and since it uses usability testing scenerios in the research, I gave it a shot.
The paper is Using meaningful and stable categories to support exploratory web search: Two formative studies by Bill Kules and Ben Shneiderman, of the Department of Computer Science, Human-Computer Interaction Laboratory and Institute for Advanced Computer Studies, University of Maryland.
The purpose of the study is to better understand how people use search engines to research topics - specifically, how categorization of search results applies to the end user experience.
"Categorizing web search results into comprehensible visual displays using meaningful and stable classifications can support user exploration, understanding, and discovery. We report on two formative studies in the domain of U.S. government web search that investigated how searchers use categorized overviews of search results for complex, exploratory search tasks."
They ran test subjects through a variety of tasks. Here is one example.
"Scenario 2 (Breast cancer) - You are a 30-year old journalist writing an article on breast cancer and what the federal government is doing about it. You are exploring the topic, starting by looking on the Web to find out what kind of information is available. You have just entered the search terms "breast cancer".For each test scenerio, three tasks were applied. An example:
"The web contains a variety of sources, perspectives and viewpoints on almost any given topic, and this is true within the federal government. Find 3 web pages providing different aspects of or perspectives on this topic. (Time limit: 3-4 minutes)"
They rated search results, including things like understandability, helpful or unhelpful, and frustrating or satisfying.
If you visit the paper, scan to section 3.7.5. Observations and participant comments to read the results. An example:
"A few participants would scan down one or two pages, and then scan up from the bottom, stating that they expected the lower-ranked results would produce different perspectives. Most participants scanned either the title only or title and snippet. Very few of these participants appeared to use the department/agency name."
It's juicy stuff for search engines, and also for web site optimizers who are curious about how search results factor into conversions, traffic and rank.
