As part of an ongoing observation of medical information sources used to retrieve information, we developed a Web portal. This portal gives access to PubMed, UpToDate, Harrison’s Online, and a Dutch pharmacotherapy database. All residents and specialists in internal medicine selecting PubMed or UpToDate from our hospital information system were automatically linked to our portal.

To enable the registration of all aspects regarding the use of PubMed, we built our own PubMed interface for accessing PubMed through e-utils [7]. E-utils gives access to full PubMed functionality. Query handling conducted by PubMed is identical to the original PubMed website, but e-utils delivers the data in XML to permit recording of the data in a database. The XML data need to be translated into Web pages to be readable for users. To mimic the functionality of PubMed, most of the special search options relevant for patient-related searches were copied in our interface: MeSH database, details, a selection of limits (publication date, publication type, human or animal, and age), and spelling. As shown in Figure 2, on the left of the page, the participant can choose to start searching for a new question, close the question, or re-open older questions (Nieuwe vraag, Vraag afsluiten, Oude vragen). There are links to background information (Achtergrond) and the manual (Handleiding). Search options are simple, advanced, details, check spelling, and MeSH database (Eenvoudig, Uitgebreid, Details, Spelling, and MeSH).

All queries were recorded as well as the use of the different search options, the articles that were selected for abstract reading, and the articles that were selected for full-text reading.

Figure 2

Screenshot of the portal: PubMed search result for “hypertension”

As the other online sources do not permit direct access to their database, we linked directly to their website. The interface of UpToDate, therefore, was presented unaltered to the physician (Figure 3). After reading the information at the website, searchers returned to our own portal to answer questions regarding their search.

Figure 3

Screenshot of the UpToDate interface (Reproduced with permission from UpToDate, Rose BD, editor, UpToDate, Waltham, MA, 2008. Copyright 2008 UpToDate, Inc. [2])

The portal was tested by direct observation using several user groups. After the testing phase, the program was introduced and tested by a select group of users during a period of 2 months. Starting October 2005, the portal was made publicly available. A manual is available from all screens in the portal. During the first year, all new users were asked if they needed help with the use of the portal. Participants received regular emails reminding them that help was available within the portal or that they could receive direct coaching.

Upon accessing the database for the first time, the physician was asked to give informed consent to the observation of the search process. The physician was also presented with background information regarding our study and was urged to read the manual, which is available from every screen of the portal.

Every search was started by entering a query and selecting an information source. Search time was recorded by the monitoring program. Sending of the first query regarding a problem was marked as the start of the search. While searching, all queries were recorded by the portal. After completing the search, participants were asked whether they found no answer, a partial answer, or a full answer to their question; answering this question marked the end of the search (Figure 4).

Figure 4

Screenshot of the page were participants could mark whether they were disturbed while searching, could select whether a complete, partial, or no answer was found, and could return to the problem

We intended to maximize the use of our computer portal. Physicians were encouraged to use the program as much as possible. At regular intervals, the database was checked to identify participants who infrequently provided details after searching. These participants were approached to determine the reason for nonresponse and were encouraged to improve their response. Nonresponse could be related to the participant but also to the monitoring system. We expected that physicians searching during daily medical care would not always be prepared to answer our questions directly after searching. Full-text articles and UpToDate were always opened in a separate pop-up window as most sites do not permit the opening of their Web pages within another frame. The Web page containing the questionnaire was available directly behind the pop-up windows. Forgetting to close the pop-up window after searching (and before closing the connection to the database) would lead to nonresponse. As both sources of nonresponse could lead to bias, we performed an additional check during the first year of our study. If participants did not fill in the questionnaire after searching, the questionnaire was repeated before the next search. As details regarding a former search are likely to become less reliable after some time, we intended to use the details provided within 24 hours after searching for a nonresponse bias analysis. After one year of monitoring, we had enough data to exclude nonresponse bias and removed the questionnaire before searching as it led to avoidance of the website.

Only problems triggered by visit rounds or related to patient contact were included in our analysis. There were four different categories of searches: (1) searches that were completed with search-related details provided in one session, (2) searches with search-related details provided during a second session within 24 hours, (3) searches with search-related details provided during a second session after 24 hours, and (4) searches with no additional information provided. To minimize the risk of recall bias, only searches of the first category were included in our study. Searches of the second category were used for nonresponse bias analysis. The last two categories were excluded. The Dutch pharmaceutical database and Harrison’s Online cannot be considered as online evidence-based information sources because they do not link the text directly to literature references. Queries sent to these databases were therefore excluded from this study.

Whether an answer is partial or complete is a subjective qualification. We therefore combined partial and full answers when determining significance of our findings. Determining statistical significance was performed by the chi-square statistic. Statistical analysis was performed using SPSS 14.0 (SPSS Inc, Chicago, IL, USA).

This study was performed in a single hospital where specialists and residents are accustomed to accessing PubMed and UpToDate as primary information sources. There are many more evidence-based information sources available on the Internet. For our observation, we chose to use the information sources that our population was familiar with, limiting the generalizability of our results.

Optimal testing of the performance of medical information sources requires taking the physician out of daily practice as physicians will not be prepared to look up answers in several databases and answer additional questionnaires during working hours. Most studies, therefore, resort to observation in laboratory situations or questionnaires without direct observation [22]. As PubMed is likely to answer most of the questions if the search time is unlimited, testing PubMed out of daily practice without time constraint is meaningless for daily care use. We used a novel approach that combined observation with post-search questionnaires. We consider PubMed and UpToDate as reliable information sources, but there is limited information that compares their usefulness in daily use. Physicians working at our hospital are very familiar with these sources; PubMed and UpToDate are therefore ideal for an observational study regarding their everyday use. There are several limitations to an observational study that apply to our study as well. We could not influence the information source approached or check whether the answer would be found in a second database in all questions. This makes a direct comparison of the information sources impossible.

We rebuilt most of the functionality of PubMed in our interface. However, exact mimicry of the website was not allowed by legal and ethical issues. Users could provide comments to the portal but did not report that the use of our interface was more difficult than the original PubMed interface.

The fact that physicians report that they have found an answer is not a guarantee that the answer really has been found. Physicians tend to overestimate the quality of the information retrieved through searching. Previous studies have shown that correct answers before searching can be incorrectly altered by searching online information sources [14,23]. Whether a partial or full answer is found is a subjective interpretation. The qualification should, however, reflect satisfaction of the participant with the obtained answer.

In many questions, the questionnaire was not filled in after searching. The major reason is opening of multiple Web pages on the screen, causing the monitoring program to disappear in the background. This, in turn, resulted in participants forgetting to answer the required information after the search within the time limit of 15 minutes. We also suspected that physicians would be reluctant to spend additional time answering search-related questions during daily care. It is likely that more complex questions leading to no answer after extensive searching will result in nonresponse. To detect whether this noncompliance would lead to a nonresponse bias, we performed a secondary analysis regarding queries answered during a second session within 24 hours. The results were comparable, showing that question complexity itself was not a reason for nonresponse.

PubMed is our default database for searching, so the use of PubMed might be overestimated. We asked whether participants were interrupted while searching, but we did not exclude these searches because we consider disturbances part of one’s daily routine. As we did not ask what database gave the answer to the question, it is impossible to identify which database contributed most to the answer when multiple sources were used. For this study, we assumed that the intention for consulting a second database was to improve the answer found in the first information source.

Our study makes a contribution in observing hospital physicians in their daily routine solving patient-related questions. We have shown that answers to questions posed during daily medical care are more likely to be answered by UpToDate than PubMed, regardless of the topic of the search. Physicians trying to answer patient-related questions use less than 5 minutes to search for an answer during daily medical care. Improving medical information sources should be aimed at delivering an answer within 5 minutes as this is the average time a hospital specialist spends finding an answer at the point of care. Future research should be aimed at comparing more information sources at different levels of the evidence pyramid. Question complexity may play a role in the choice of where to enter the hierarchy of evidence-based sources. Analysis of query content and the search process should reveal more information to improve PubMed as a search tool for daily medical care.