Searching for information on the Web to support decision making is now an important part of clinician practice [1]. While much research focuses on the design of retrieval algorithms to identify potentially relevant documents, there has been little examination of the way that different search engine capabilities influence search behavior. Yet, to develop information retrieval systems that actively support decision making, it is necessary to understand the complex process of how people search for and review information when making decisions [2] and to design search user interfaces appropriate for these needs.

Recent studies of clinical search strategies have concentrated on methods of optimizing queries sent to information retrieval systems that enhance the performance of the retrieval. Hoogendam and colleagues conducted a prospective observational study of how physicians at a hospital used PubMed to search for information during their daily clinical activities [3]. They found that the likelihood of physicians viewing article abstracts returned from PubMed increased as the number of terms contained in a search query increased. Haase and colleagues investigated the optimal performance for different search engines in retrieving clinical practice guidelines by combining different search query terms [4]. Our own prior analysis of information searching by clinicians used a Bayesian belief revision framework to retrospectively model how documents might influence decisions during and after a search session [5]; the analysis demonstrated that clinicians can experience cognitive biases while searching for online information to answer clinical questions [6].

Few studies have looked at how clinicians reformulate queries and select sources to retrieve information during a search session to answer clinical questions. In previous studies, we have shown that a task-based search engine design allows for faster clinical decision making (ie, “decision velocity”) compared with purely resource-based engines at no cost in correctness of answers [7]. Similar results with respect to search times have been noted by others for the use of topic-specific “infobuttons” [8]. In the current study, we sought to understand the basis for these performance variations, by testing whether differences in search engine interface design are associated with any differences in user search behaviors.

Of 75 clinicians, 39 were randomly allocated to use of the resource-based system and 36 to use of the task-based system. Two resource-based scenarios were not completed, giving a total of 310 (ie, 39×8−2) search sessions, 1708 searches, and 1455 document accesses using the resource-based system. The task-based system generated 288 (ie, 36×8) search sessions, 873 searches, and 1136 document accesses.

Clinicians using the resource-based system appeared to favor a “breadth-first” search strategy, exploring different resources with the same keywords in the query before searching in a specific resource with query reformulations. Clinicians using the task-based system were provided with results from multiple resources in each search and so appeared to favor a “depth-first” search strategy, searching in the same task profile exhaustively with different keyword reformulations in the query before moving to other profiles.

We have previously shown that changes in search engine design and interface were associated with changes in clinical decision velocity, number of search actions undertaken, and ultimate decision outcome [7]. To understand the basis for such differences, we have now looked at the type of actions undertaken by users of two different systems and the sequences of these actions. While it was the intention of the experiment to detect changes in search behavior, our present analysis extends the analytic framework of the original experiments and may thus suffer from being a post hoc explanation of the observed differences. This limitation may readily be addressed by further experiments specifically designed to test for changes in search strategy.

Further study is needed to understand how clinicians assess the results of a search and formulate the next step in their strategy. We have discussed elsewhere that the process of searching can be thought of as a conversation [15] where individuals ask questions of knowledgeable agents (eg, information retrieval systems or people) to help find answers to their questions. Thinking of the interaction with a search engine as a conversation between a human with a question and a search engine with capabilities to help find an answer may help us understand the human behaviors observed in this study.

According to Grice’s conversational maxims [15], (which were originally created to describe the “rules” for effective human conversations), an answer to a question may be inappropriate for a number of reasons. The respondent may be poorly qualified to answer the question (eg, the respondent may be an inappropriate, out of date, or otherwise misleading information source); may misunderstand the question (eg, the query may not be well expressed in terms understandable by the resource); or may reply with unhelpful or irrelevant information (eg, because of poor relevance metrics of the search algorithm). We can speculate that the search actions taken by clinicians are in response to judgments they make about the progress of their “conversation” with the information retrieval system.

One can hypothesize, when clinicians are faced with a choice of several resources with no clear indication of which is the best, they scan multiple resources to gauge the "competence" of each before committing to a detailed conversation with the resource they feel best qualified to help. In contrast, clinicians with a task-based system are simultaneously receiving answers from multiple resources and so should be able to quickly form a view of the overall capabilities of the group of resources being simultaneously searched. Not faced with concerns about the competence of the system they are interacting with, clinicians focus on improving the dialogue with the system. This is done by finding different ways to ask the same question or by changing the question focus if there has been a “misunderstanding.” As a result, this could explain why users of task-based systems conduct fewer searches and consult fewer documents [7], that is, these users may not need to credential the resources they are interacting with in the same way that users of resource-based systems appear to do.

Overall, given the clear differences in the styles of user-system dialogue demonstrated in this study, and the impact of such behavior on the clinical utility of information retrieval systems, discovering ways of optimizing the dialogue between knowledge sources and users seems a productive line of further enquiry.