The ubiquity of internet-connected smart devices in the clinical setting [1] and the availability of preappraised point-of-care evidence summaries [2,3] have changed how health care professionals seek, consume, and implement information in the care pathway [4]. For instance, a generation now separates the sample of health care professionals who participated in the seminal work of Covell et al [5] in 1985 and their digitally native descendants. That sample (47 medical physicians) reported having 2 clinical questions for every 3 patient visits and preferred textbooks, journals, and drug information indexes to fulfill their information needs [5]. Importantly, these findings are now irreconcilable with modern clinical archetypes as “computered sources were reported to be used least often” [5].

Since then, the delivery of clinically relevant information to the care pathway has progressed to smart devices capable of running stand-alone software apps. At present, an increasing proportion of health care professionals use these devices to inform their practice [1]. This change has been charted in thousands of studies [6] and syntheses of studies [7,8], which have demonstrated the use of web-connected smart devices for fulfilling clinical information needs [5,8-11]

Unfortunately, the current understanding of information-seeking and utilization behaviors is shaped by suboptimal empirical models. Self-report questionnaires [5,9], interviews [12], and think-aloud [13-15] methodologies may offer a feasible way to estimate how health care professionals perceive their own information-seeking behaviors, their frequency, the topics to which they relate, the resources they use to fulfill these needs, and the strategies they employ to harvest information from these resources. However, these behaviors can now be more directly observed using web server log file analysis [16-18].

To date, researchers have leveraged web logs to evaluate usage patterns for specific websites such as Wikipedia [19,20], PubMed [21,22], UpToDate [23], and web-based institutional health science libraries [16,17]. Typically reported usage metadata include the number of time- or user-dependent hits or sessions on the website [24,25], the most frequently searched topics [18,22], and the number of click-throughs from one section of the website to another [18].

Importantly, these analyses tend to be constrained to individual websites [22] or institutional portals [23], which redirect to a limited group of external websites. These constraints undermine the validity of modern information-seeking paradigms and bely our understanding of the specific digital information resources that are used in an unconstrained web environment to fulfil information needs. Health care professionals in the real world have unbounded web access; this does not reflect the sandboxes of preselected information resources that are evaluated in the available body of research. Furthermore, little work in the web log analysis literature has focused on how these resources can be used to answer clinical questions.

Therefore, the aim of this study is to evaluate the use of web-based resources for fulfilling clinical information needs. Specifically, our objectives were to conduct a randomized crossover trial, whereby a group of physiotherapists were subjected to a multiple-choice questionnaire (MCQ) examination, which they completed under 3 conditions: (1) assisted via a web browser with unconstrained web access, (2) assisted via a federated search engine “portal” app, and (3) unassisted.

By including both a federated search system and unconstrained web use in a single study design, we sought to evaluate the potentially mediating effects of the access tool on the kinds of resources being used and the time spent doing so. This study aims to address both hypothesis-confirming and hypothesis-generating research questions.

The confirmatory hypotheses were as follows:

The hypotheses that this research may generate relate to the following:

Ethical approval for this randomized crossover trial was granted by the affiliate review board of the institution at which the authors were based (reference: LS-18-25). Study design, conduct, analysis, and results are reported according to the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist.

Prospective participants included practicing physiotherapists who were recruited with the help of collaborating health care institutions. The study was advertised on the websites of these institutions, and a notice was sent throughout the recruitment period to prospective participants via the “staff bulletin” of the respective institutions. Physiotherapists were chosen for evaluation, as their information behavior has not been well addressed in the research literature to date, despite their prominent role in health care [27].

All individuals met the following inclusion criteria: adults (>18 years of age), currently employed by collaborating health-related institutions and involved with the management and treatment of patients, provided informed consent (which was obtained digitally during authentication with the federated search app).

Information needs were induced via the administration of an MCQ examination completed at 3 time points under 3 conditions in random order: (1) assisted by a standard web browser, (2) assisted by a federated search engine, and (3) unassisted.

One investigator (CD) organized an appropriate time and location to conduct all test sessions; to maximize participant retention, testing was conducted at a time and place that suited participants. Short-term rescheduling (ie, within 1 week of the original designated test time) was facilitated in the event of unexpected delays or schedule conflicts.

For the purpose of this study, we defined an information need in the assisted conditions as any instance in which a participant used an assistive technology to access a digital information resource to inform their choice of answer in the MCQ. In the “assisted by a web browser” condition, participants were provided with a web-connected laptop preinstalled with a web browser (Google Chrome). For the “assisted by a federated search engine” condition, participants were required to install and register an account with a free experimental federated search engine called SciScanner [28] on their smartphone.

All web and federated search log files were accrued in a single database after participants had completed the test protocol for each “assisted” condition and were filtered to remove setup (eg, log-in events to institutional networks) and duplicate events. All log files were then grouped into sessions based on the timestamps of the first and last log, which were cross-referenced with the recorded date, start time, and duration of the examination. Each session was then manually segmented by the primary author into individual information-seeking events based on search queries and the timestamps of every event; consecutive search queries that were deemed to be thematically related (eg, “sensation to medial tibia,” “nerve supply to medial leg,” and “saphenous nerve”) and those that occurred within a similar timeframe were grouped as a single information-seeking event. Information-seeking events were then partitioned into 3 periods: primary, intermediate, and terminal. The primary period related to the first search that was logged by participants in an information-seeking session or the first search logged in a thematically similar series of consecutive search segments. Terminal periods were defined as the last search logged by participants in an information-seeking session or the last search logged in a thematically similar series of consecutive search segments. Intermediate periods included consecutive logs between the primary and terminal periods.

For each period of an information-seeking event, the resources that were accessed were identified. This was automatically recorded in the federated search logs by design and was determined based on the website URL for the web logs. The number of hits and the accumulated time for each hit were then determined for each resource for the entire test cohort. Segmentation, partitioning, and preparation for data analysis are displayed in Figure 2.

Previous research on knowledge acquisition using web-based resources among medical trainees identified a 15% difference between assisted and nonassisted assessment scores (Cohen d=0.86) [21,26]. On this basis, a sample of 18 pairs of physiotherapists was deemed sufficient to achieve experimental power. To account for a 5% (2 of 36) potential dropout, 38 participants (19 pairs) were recruited; however, there were no dropouts, so all participants were included in the analysis.

We adopted a 3-tiered analysis paradigm for our outcomes. The first tier was related to the MCQ results (examination-level analysis), the second to the individual questions of the MCQ (question-level analysis), and the third was related to the log files recorded in instances where there was an information need (log-level analysis; assisted conditions only).

The demographics of the participating sample were represented using descriptive statistics.

A total of 38 physiotherapists fully participated in the study, completing an examination under each of the 3 experimental conditions in random order using Vickers’ block randomization [42]. The demographic characteristics of the included sample are presented in Table 1. All tests were conducted in the 6-month period from June 2019 to January 2020. All participants completed all 3 conditions within a 1-month (4 week) period of enrolling in the study. A CONSORT diagram is presented in Figure 3 [43].

The examination-level GEE estimated a main effect for condition (P<.001). Examination scores differed significantly between the unassisted condition (mean examination score 47.11%, SD 12.7) and both assisted conditions (mean federated search condition 57.5%, SD 12.8, P=.001; web search condition 63.3%, SD 12%, P=.02). The mean examination score and examination duration for the 3 experimental conditions, and the mean number of information needs for the assisted conditions are presented in Table 2.

Participants experienced 845 information needs out of a total of 1520 questions, with a rate of 55.6% (assisted conditions only). The question-level GEE was used to further investigate the relationship between individual information needs and answers (correct and incorrect). This GEE estimated a main effect for condition (P<.001) and the presence of an information need (P=.009). An analysis of the parameter estimates associated with these main effects revealed that participants were more likely to answer correctly in both the federated search condition (P=.003) and the web browser condition (P<.001) compared with the unassisted condition.

Despite the observation of a main effect for the presence of an information need, there were no significant differences based on the parameter estimates at the level of P<.05 in the rate of answering individual questions correctly in the presence or absence of an information need (P=.06).

The average rate of answering correctly for each condition in the absence or presence of an information need is presented in Table 3.

The 845 information needs corresponded to 1987 web logs and 3277 app logs. This data set of app and web logs comprised all participants’ information-seeking sessions, which themselves included multiple information-seeking events (or information trails) split into primary, intermediate, and terminal periods.

Table 4 presents the most popular resources used during the primary, intermediate, and terminal periods in the web and app log conditions.

In the web-assisted condition, participants spent a total of 00:45:25 min (383 hits) in the primary period, 04:14:55 hours in the intermediate period (1356 hits), and 06:26:43 hours in the terminal period (248 hits). This corresponded with a median time per resource of 1 second in the primary period, 4 seconds per resource in the intermediate period, and 86 seconds per resource in the terminal period.

In the federated search condition, participants spent a total of 01:47:21 hours (486 hits) in the primary period, 08:42:24 hours in the intermediate period (2076 hits), and 04:24:06 hours in the terminal period (668 hits). This corresponded with a median time per resource of 3 seconds in the primary period, 4 seconds per resource in the intermediate period, and 5 seconds per resource in the terminal period.

The log GEE estimated a main effect for condition and for the period of the information trail (P<.001). There was also a significant main effect for the interaction between condition and period (P<.001). An analysis of the parameter estimates associated with these main effects revealed that participants spent less time per hit in the primary and intermediate periods for the web search condition compared with the federated search condition, and more time in the terminal period in the web search condition than in the federated search condition. The median and mean values for each condition stratified by search period are presented in Table 5.

Despite the methodological strengths of the crossover experimental design and the insights garnered via the use of web log analysis to determine what digital information resources were used to fulfill information needs and the novelty of evaluating these outcomes among physiotherapists, this study is not without limitations. First, an MCQ was used as a surrogate stimulus for information needs and may not be an accurate representation of the clinical setting. The validity of the MCQ questions as a measure of physiotherapy knowledge and by extension, health care delivery, was assumed on the basis that they were from the examinations of recognized accreditation bodies. It must be acknowledged that although these questions are used to assess physiotherapists’ capacity to practice, they may not actually represent the information needs of the authentic clinical encounter. Second, this study does not address an enduring question as to whether the fulfillment of clinical information needs alters treatment practices and patients’ outcomes; it has always been assumed, but never proven [46], that the translation of the latest high-quality research evidence to the care pathway optimizes clinical practice behaviors and improves patients’ outcomes. Finally, although the use of a federated search portal system provides a useful comparator for unconstrained web browsing in evidence search and retrieval patterns and their effect on MCQ examination scores, the results presented in this study for this portal system (ie, SciScanner) are unique to it and have limited external applicability for other portal systems.

On the basis of an MCQ examination protocol, we identified that physiotherapists experienced an information need in 55.59% (845/1520) of theoretical questions and, when they were provided with access to digital information resources accessed via the web or a federated search software app, the fulfillment of these needs was associated with improved rates of answering examination questions correctly. The physiotherapists in this study exhibited a high preference for Google as both a directory and a resource, with Wikipedia and PubMed being the next most popular resources. The implications of relying heavily on Google as a search and retrieval mechanism for health-related information warrants further investigation, whereas the emulation of these findings in an authentic clinical setting would be an important research pursuit in the future.