This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Background
Conventional Web-based search engines may be unusable by individuals with low health literacy for finding health-related information, thus precluding their use by this population.
Objective
We describe a conversational search engine interface designed to allow individuals with low health and computer literacy identify and learn about clinical trials on the Internet.
Methods
A randomized trial involving 89 participants compared the conversational search engine interface (n=43) to the existing conventional keyword- and facet-based search engine interface (n=46) for the National Cancer Institute Clinical Trials database. Each participant performed 2 tasks: finding a clinical trial for themselves and finding a trial that met prespecified criteria.
Results
Results indicated that all participants were more satisfied with the conversational interface based on 7-point self-reported satisfaction ratings (task 1: mean 4.9, SD 1.8 vs mean 3.2, SD 1.8, P<.001; task 2: mean 4.8, SD 1.9 vs mean 3.2, SD 1.7, P<.001) compared to the conventional Web form-based interface. All participants also rated the trials they found as better meeting their search criteria, based on 7-point self-reported scales (task 1: mean 3.7, SD 1.6 vs mean 2.7, SD 1.8, P=.01; task 2: mean 4.8, SD 1.7 vs mean 3.4, SD 1.9, P<.01). Participants with low health literacy failed to find any trials that satisfied the prespecified criteria for task 2 using the conventional search engine interface, whereas 36% (5/14) were successful at this task using the conversational interface (P=.05).
Conclusions
Conversational agents can be used to improve accessibility to Web-based searches in general and clinical trials in particular, and can help decrease recruitment bias against disadvantaged populations.
embodied conversational agentsearch user interfaceinformation retrieval user interfaceWeb searchhealth literacyrelational agentcomputer literacysearch engineInternetIntroduction
The majority of US adults look online for health information [1,2]. However, disparities in the use of the Internet for finding health information remain [3,4]. One specific cause of these disparities may be that keyword-based search engines such as Google—although the primary search portals for most users—may actually represent a significant barrier for many disadvantaged individuals. Prior research has demonstrated that people with low health literacy, the ability to acquire and act on information related to health care [5,6], have particular difficulty using keyword-based search interfaces. Agree et al [3] demonstrated that individuals with low health literacy had lower success rates when using these interfaces to search for general health information on the Web. Usability by people with low health literacy is important because this population comprises 36% of US adults [5].
In addition to general-purpose search engines, many search engines and interfaces have been developed for specific kinds of health care information. One example is the clinical trial search engine, which retrieves descriptions of clinical trials from a repository or database [7]. Several of these search engines are available on the Web, developed by both commercial firms and the US government (eg, the National Cancer Institute [8]). Individuals use these search engines to find trials for which they may be eligible and in which they may be interested in participating. Utami et al [9] found that individuals with low health literacy found fewer clinical trials and took longer to complete standardized search tasks using a Web-based clinical trial search engine compared to those with adequate health literacy. Usability of clinical trial search engines by people with low health literacy is especially important because there is a disproportionate representation of minorities in this group [10-12] leading to reduced access for disadvantaged populations to information about clinical trials. Although Web-based clinical trial search engines hold the promise of providing universal access to information, conventional search systems may further promote disparities in clinical trial recruitment by catering primarily to populations of well-educated individuals with high levels of health and computer literacy.
Conventional Web form-based search engine user interfaces (eg, Google) typically make exclusive use of user-supplied keywords, whereas others combine keyword input with multiple-choice options, referred to as “facet-based” search interfaces [13]. Several prior studies have investigated the use of these search interfaces for users with low domain knowledge [14], who speak a language that is different than that of the Web form [15], who are children [16], or older adults [17], all who share characteristics with our task and population. These studies have demonstrated that even the simplest keyword-based search interfaces are unusable for many users and that special design considerations—such as simplifying results [17] and providing language and interaction support [16]—are important for disadvantaged users. Users may also be influenced by contextual cues when evaluating results from search engines [18] and those with low health literacy may be particularly susceptible to these cues when evaluating search results, relying on such features as position in search results, quality of pictures, and celebrity endorsements [19].
In this paper, we describe the design of a Web-based clinical trial search engine that we designed to mitigate barriers associated with low health literacy. The search task is framed as a conversation with an animated character to make it as familiar and approachable as possible, and a number of additional features and simplifications were made to help users with low health literacy navigate the overall clinical trial search process. We conducted a randomized trial, comparing the conversational search engine to an existing conventional Web-based search engine.
Design of the Conversational Search Engine
The overall task the conversational search engine supports is finding one or more cancer-related clinical trials for which the user is eligible, based on initial demographic criteria and in which the user is provisionally interested, using publically available information. The search engine indexes trials from the National Cancer Institute’s (NCI) database of more than 10,000 active trials (at the time this work was conducted) [8].
Based on our experience in developing several health counseling dialog systems for patients with low health literacy [20], previous studies demonstrating greater user recall with audiovisual information combined with conversational style [21], and with animation combined with speech compared to text [22], we designed the overall interaction as a dialog with an embodied conversational agent [23]. The agent speaks using synthetic speech, generated from an augmented transition network-based dialog engine [24], template-based text generation [25], and a dynamically updated user model accompanied by conversational nonverbal behavior (eg, hand gestures, facial displays, gaze) animated in synchrony with the speech [26] (Figure 1). The agent also manipulates artifacts it is discussing with the user; in this case, documents that represent aspects of the clinical trials being discussed. User inputs to the conversation are restricted to multiple-choice selection of utterances from a list that is dynamically updated during each turn of the conversation. Thus, the interaction is system-initiated at the dialog adjacency-pair level (eg, agent question / user response), but user initiative is provided by allowing the user to select topics of conversation and ask questions at predefined points in the dialog by selecting from predefined lists. We have successfully used this interface modality with more than a thousand patients in clinical trials, including hundreds who have low health literacy and many who have never touched a computer before [20,27]. The resulting system could be characterized as a “fully faceted” search interface in which users are never asked to recall and type text, but are always scaffolded with the range of possible inputs they can make [13].
The overall search experience is framed as an extended conversation, in which the user is first interviewed about their requirements and preferences and then shown candidate trials with the agent providing as much scaffolding—through tutorials, explanations, and suggestions—along the way as possible. Given that clinical trial descriptions can be very complex and tedious for users to read, we err on the side of eliciting as much information as possible from users before the search in an attempt to identify trials that are most fitting. In addition, we designed the system to display information about a trial in stages, revealing only the details a user needs at each point of their evaluation. The overall flow of a typical conversation is shown in Figure 2.
To define the search criteria the agent elicits from the user, we leveraged qualitative findings from our usability study [9]. Participants in this study were asked to choose between pairs of clinical trial descriptions and then asked to explain their rationale. Analysis of explanations using grounded theory [28] revealed information-seeking practices and deliberation themes. We cross-referenced the resulting list of search criteria preferences elicited from users in this study with the clinical trial schema in the NCI database. We found that some user criteria already existed as database indexes, including participant age, sex, cancer type, study geographic location, trial type and phase, and the use of an investigational drug. Additional user criteria did not exist as database indexes, but could be inferred through text classification of text fields in the database, including subjective assessment of the likelihood that study participation would involve painful procedures, subjective assessment of protocol invasiveness (eg, survey vs diagnostic vs treatment), and overall time commitment. These inferred criteria (pain, invasiveness, and time burden) were computed for each trial based on a decision tree algorithm (ID3 [29]) that used word occurrence features in the trial description text trained on hand-rated examples. In our runtime clinical trial search algorithm, the search criteria that could be mapped to existing database indexes are used to search the database, whereas the criteria inferred via text classification are used to sort results.
In addition to the overall structure of the interaction shown in Figure 2, we included several features in the search user interface to assist users with low literacy in their search:
Dictionary: the NCI website provides users with a dictionary of medical terms; however, this dictionary is available on the NCI site as a separate module from the search engine. In our user studies of this website, we observed that finding definitions often distracted users from their main search task. We integrated a dictionary with our search functionality; although the agent explains search results to users, the character automatically extracts difficult terms from the text and offers to explain them.
Read aloud: users are able to ask the agent to read aloud, and repeat if necessary, any clinical trial text or definition. While speaking aloud, the agent holds up a visualization of the text, enabling users to read along.
Simplified title: clinical trial titles can be very long, complex, and hard to remember. We simplified display titles using the phase and type of the trial, and cancer type the trial is for (eg, replacing “A Study of BFTZ2252 Given With Patanobib or Dopepaxel in Subjects With Non-Small Cell Lung Cancer” with “Phase II Treatment Trial for Lung Cancer”).
Education modules: the agent offers to explain several concepts underlying clinical trials, such as voluntariness and risk, at appropriate points in the search.
User search criteria confirmation: the agent periodically displays the user’s search criteria and allows them to revise their choices. This is done after the initial interview, after each matching clinical trial is discussed, and whenever the agent runs out of trials to show the user.
Query refinement: if a search returns no results or the user exhausts the list of indexed trials, the agent suggests ways to modify the search criteria.
Bookmarking: users can save references to clinical trials for review in subsequent search sessions.
Summary of views: the agent periodically displays a list of trials viewed during the search session with bookmarked trials highlighted.
Levels of detail: each clinical trial description is presented in 3 levels of detail with associated displays. Users are initially only shown the title and eligibility criteria for a trial (see Figure 1), but are then given the option to view the trial research purpose, after which they are given the option of viewing details of the trial procedures.
Conversational agent search interface.
Typical dialog flow in a search session.
Study Aims
The overall aim of this work is to develop a Web-based search interface that is more usable by individuals with low health and computer literacy. We hypothesize that the conversational search interface will lead to greater search successes and higher levels of satisfaction compared to conventional keyword- and facet-based search engines for all users, but that the differences will be especially pronounced for individuals with low health literacy.
Methods
To evaluate our system, we conducted a between-subjects randomized trial comparing our conversational agent search engine (“agent”) to the conventional facet- and keyword-based search engine (“control”) developed by the NCI ([7]) with both search engine interfaces indexing the same set of clinical trials. Participants were recruited from a pool of adult English-speaking cancer patients from across the literacy spectrum. Participants already had sociodemographic measures recorded and health literacy assessments completed for a prior study at Virginia Commonwealth University. The study protocol was approved by the Boston Medical Center and Virginia Commonwealth University IRBs and informed consent was obtained from all study participants. Health literacy was assessed using the Rapid Estimate of Adult Literacy in Medicine (REALM) [30]. Participants were split into adequate and inadequate health literacy groups using a REALM score of 9th grade as a cut-off as other authors have done [31-34].
We asked participants to perform 2 search tasks. In task 1, participants were asked to search for a clinical trial for which they would be provisionally eligible and in which they would be interested. In task 2, they were asked to search for a trial for someone else with specified eligibility criteria (ie, age, cancer type, trial type, geographic location) as a standardized test so that we could determine whether any trials they found actually matched the specified criteria (Textbox 1).
To ensure accessibility for participants with low computer and health literacy, we designed the experiment so that they were able to do the study either in the laboratory (for those without access to computers) or at home. The experiment software first gave participants a short tutorial on using the system, including a practice task. The first search task was then displayed. To complete each task, participants were redirected onto another Web page that had the agent or the NCI search engine (Figure 3). If the participant found a trial, they entered the clinical trial ID number into a text field and clicked an “I found a trial” button. If they could not find a trial, they clicked on an “I cannot find a trial” button. As soon as users completed each task, they were prompted to fill in a Web form questionnaire that captured the study measures. At the completion of each task, the first 7 questions in Table 1 were automatically administered via Web forms. Participants were also asked to recall the number of trials they examined and the number of these that met their criteria after each task. The Web server also captured the clinical trial ID that the participants found (if any) and the time needed to complete each task. At the completion of both tasks, the remaining 5 questions (questions 8-12 in Table 1) were administered verbally by a research assistant (for those conducting the study in person) or over the phone (for those conducting the study remotely). Participants completing the study at home did so on their own, without online assistance, and were called within 36 hours of completing the online tasks to obtain final outcome measures. Nonparametric statistics (chi-square tests for frequencies and Mann-Whitney U tests for all other measures) were used given the nature of the data and nonnormality of most distributions.
Standardized search task (task 2).
Now here is your second task. Please write it down.
This is Rosa. She is a cancer patient.
<IMAGE OF ROSA>
Here is some information about Rosa:
Age: 70 years old.
Cancer type: Breast cancer.
Trial type she would like: Treatment trial.
Location of trial she would like: Can be anywhere.
We would like you to use the information above to find a clinical trial for Rosa.
Once you find a trial, please enter its ID number into the box on the bottom right of the screen where it says "TRIAL ID” and click the button that says "I found a trial.” Also, please write down the protocol ID number on a piece of paper.
If you have spent some time looking but do not think you can find a trial, then click on the button that says "I cannot find a trial” at the bottom left corner of the screen.
If you are ready to begin, click the "I am ready" button.
Experimental setup for control condition.
Self-report measures used in the study.
Scale self-report item
Anchor 1
Anchor 7
1. To what degree did you know what you wanted in a trial? (task 1)
Not at all
Exactly
2. To what degree did the trial match what you were looking for?
Not at all
Exactly
3. How likely are you to sign up for the trial that you found? (task 1)
Not likely
Very likely
4. How much time do you feel it took to use the system?
Too little
Too much
5. How satisfied were you with the clinical trial search system?
Not at all
Very
6. How frustrated do you feel right now?
Not at all
Very
7. How pleased do you feel right now?
Not at all
Very
8. How much pressure did you feel to volunteer for a trial?
No pressure
A lot of pressure
9. How much information do you feel was presented by the system?
Not enough
Too much
10. How likely would you be to use the system again, if you wanted to find another trial?
Not likely
Very likely
11. How likely would you be to recommend the system to someone else who was looking for a trial?
Not likely
Very likely
12. How much do you trust the information you received from the system?
Not at all
Very much
Results
Participant sociodemographic information is shown in Table 2. A total of 89 individuals participated; mean age was 59.2 (SD 9.8) years, 46% (48/89) were female, and 27% (23/89) had low health literacy. A current cancer diagnosis was reported by 98% (87/89) of participants: 32% (28/89) with hematologic cancer, 14% (12/89) with breast cancer, 14% (12/89) with genitourinary cancer, 14% (12/89) with head and neck cancer, and 10% (9/89) with lung cancer. Most (70%, 62/89) reported regular computer use and regular use of Web-based search engines (52%, 46/89). Although only 21% (19/89) reported previous participation in a cancer-related clinical trial, 52% (46/89) expressed interest in participating in one. Approximately half of the participants (48%, 43/89) were randomized to the agent condition. Of the 89 participants, 53 (60%) conducted the study in the laboratory and 36 (40%) conducted the study over the Web at home. A few participants could not complete some of the tasks (17%, 14/89) due to technical or other problems. The primary study results are shown in Table 3.
Task 1 Results for All Participants
In the initial task, participants were asked to find a clinical trial for themselves. Most participants started this task without a clear idea of what they were looking for and rated the degree they knew what they wanted in a trial a mean 2.8 (SD 1.9) on a scale of 1 to 7. Nevertheless, 45% (19/42) in the agent group and 31% (14/45) in the control group successfully found a trial; there was no significant difference between groups (χ21=1.8, P=.52). The degree to which participants felt these trials matched what they were looking for was significantly greater in the agent condition compared to the control condition (mean 3.7, SD 1.6 vs mean 2.7, SD 1.8, U=465, P=.01). Participants were significantly more satisfied with the agent compared to the conventional interface (rating mean 4.9, SD 1.8 vs mean 3.2, SD 1.8, U=363, P<.001) and felt significantly less frustrated (rating mean 2.1, SD 1.7 vs mean 3.7, SD 2.2, U=405, P<.001) and more pleased (rating mean 5.1, SD 2.1 vs mean 3.4, SD 1.9, U=380, P<.001) with the agent after completing the task compared to those in the control condition.
There were no significant differences between results for those who completed the study in person versus at home.
Task 2 Results for All Participants
In the second task, participants were asked to find a clinical trial that satisfied a prespecified set of criteria as a standardized task. Although 48% (20/42) in the agent group and 40% (18/45) in the control group claimed to find trials that met the criteria (χ21=0.5, P=.52), only 43% (18/42) and 31% (14/45), respectively, actually found a correct trial (χ21=1.3, P=.28). However, participants in the agent group felt that the trials they found matched the criteria to a greater degree compared to those in the control group (mean 4.8, SD 1.7, vs 3.4, SD 1.9, U=381, P<.001). As with task 1, participants in the agent group were significantly more satisfied (rating mean 4.8, SD 1.9 vs mean 3.2, SD 1.7, U=336, P<.001) and pleased (rating mean 4.6, SD 1.8 vs mean 3.1, SD 1.7, U=358, P<.001), and significantly less frustrated (rating mean 2.6, SD 1.9 vs mean 3.8, SD 2.2, U=429, P=.01) after completing their task compared to those in the control group.
As with task 1, searching with the agent tended to take longer compared to the conventional interface (mean 8.2, SD 5.3 minutes vs mean 6.4, SD 4.3 minutes), but this did not meet statistical significance (U=507, P=.06). However, participants felt the agent took significantly less time compared to the conventional interface (mean 4.2, SD 1.1 vs 5.1, SD 1.7, U=466, P=.03).
Participant sociodemographics.
Variable
All
N=89
Agent
n=43
Control
n=46
P
Sex (female), n (%)
48 (54)
18 (42)
30 (65)
.03
Age (years), mean (SD)
59.2 (9.8)
58.6
59.7
.59
Race, n (%)
.45
Black
41 (46)
19 (44)
22 (48)
White
48 (54)
24 (56)
24 (52)
Other
0 (0)
0 (0)
0 (0)
Hispanic or Latino, n (%)
0 (0)
0 (0)
0 (0)
N/A
Education, n (%)
.38
<High school
15 (17)
9 (21)
6 (13)
High school
10 (11)
6 (14)
4 (9)
>High school
62 (71)
27 (63)
35 (78)
Married, n (%)
40 (45)
16 (32)
24 (52)
.20
Health literacy (REALM Score)
Mean (SD)
57 (15)
56 (16)
59 (15)
.35
Adequate (≥60), n (%)
65 (73)
27 (64)
38 (83)
.05
Study location (in person), n (%)
53 (60)
27 (63)
26 (57)
.67
Computer experience, n (%)
.64
Never used one
7 (8)
5 (12)
2 (4)
Tried one
21 (24)
10 (24)
11 (24)
Use regularly
53 (61)
24 (57)
29 (64)
Expert
6 (7)
3 (7)
3 (7)
Search engine experience, n (%)
.99
Never used one
15 (17)
7 (17)
8 (18)
Tried one
16 (18)
8 (19)
8 (18)
Use regularly
45 (52)
22 (52)
23 (51)
Expert
11 (13)
5 (12)
6 (13)
Clinical trials knowledge, n (%)
.12
None
16 (18)
12 (29)
4 (9)
A little
40 (46)
16 (38)
24 (53)
Fair amount
29 (33)
13 (31)
16 (36)
Expert
2 (2)
1 (2)
1 (2)
Participated in cancer clinical trial before (yes), n (%)
18 (21)
7 (17)
11 (24)
.43
Actually interested in participating in a trial now? (yes), n (%)
44 (52)
23 (56)
21 (48)
.52
To what degree do you know what you want in a trial?a mean (SD)
2.8 (1.9)
2.7 (1.7)
3.0 (2.1)
.60
a Anchor 1=I didn’t know at all; anchor 7=I knew exactly.
Results for Low Health Literacy Participants
Table 4 shows the results by study condition for the 24 participants with low health literacy. The results are very similar to those for all study participants (Table 3), with one notable exception: in the standardized task (task 2), none of the low literacy participants were able to find a clinical trial that met the given criteria using the conventional interface. However, 36% (5/14) of low literacy participants were able to find a correct clinical trial using the agent. This difference was near significant (χ21=3.7, P=.05).
Primary study results.
Measure
Task 1
Task 2
Agent
(n=43)
Control
(n=46)
P
Agent
(n=43)
Control
(n=46)
P
Completed task, n (%)a
37 (86)
37 (80)
.58
36 (84)
37 (80)
.79
Declared found a trial, n (%)a
19/42 (45)
14/45 (31)
.19
20/42 (48)
14/45 (40)
.52
Found a correct trial, n (%)a
—b
—b
—b
18/42 (43)
14/45 (31)
.28
Elapsed time (minutes), mean (SD)c
12.6 (9.2)
9.0 (8.4)
.06
8.15 (5.3)
6.4 (4.9)
.06
Number of trials examined (self-report), mean (SD)c
2.8 (3.0)
3.8 (6.7)
.56
3.0 (3.1)
4.9 (9.4)
.54
Trials examined that meet criteria (self-report; % of trials examined), mean (SD)c,d
56 (39)
34 (35)
.06
64 (37)
44 (41)
.09
To what degree did the trial match what you were looking for? (range 1-7),c mean (SD)
3.7 (1.8)
2.7 (1.6)
.01
4.8 (1.7)
3.4 (1.9)
.00
How likely are you to sign up for the trial that you found? (range 1-7), mean (SD)c
3.3 (1.7)
2.9 (1.7)
.21
—e
—e
—e
How much time do you feel it took to use the system? (range 1-7), mean (SD)c
4.3 (1.3)
4.6 (1.8)
.61
4.2 (1.1)
5.1 (1.7)
.03
How satisfied were you with the clinical trial search system? (range 1-7), mean (SD)c
4.9 (1.8)
3.2 (1.8)
<.001
4.8 (1.9)
3.2 (1.7)
<.001
How frustrated do you feel right now? (range 1-7), mean (SD)c
2.1 (1.7)
3.7 (2.2)
.001
2.6 (1.9)
3.8 (2.2)
.01
How pleased do you feel right now? (range 1-7), mean (SD)c
5.1 (2.1)
3.4 (1.9)
.001
4.6 (1.8)
3.1 (1.7)
.001
How much pressure did you feel to volunteer for a trial? (range 1-7), mean (SD)c
1.2 (0.6)
1.4 (1.1)
.99
—e
—e
—e
How much information do you feel was presented by the system? (range 1-7), mean (SD)c
4.2 (1.7)
4.3 (1.8)
.98
—e
—e
—e
How likely would you be to use the system again, if you wanted to find another trial? (range 1-7), mean (SD)c
5.0 (2.1)
4.1 (2.4)
.07
—e
—e
—e
How likely would you be to recommend the system to someone else who was looking for a trial? (range 1-7), mean (SD)c
5.1 (2.2)
4.5 (2.5)
.25
—e
—e
—e
How much do you trust the information you received from the system? (range 1-7), mean (SD)c
5.7 (1.6)
5.1 (1.9)
.13
—e
—e
—e
a Chi-square test.
b Task 1 involved participants finding trials they were interested in, so there was no way to objectively assess whether the trials they found were “correct”.
c Mann-Whitney U test.
d Trials examined that meet criteria was a subjective self-report measure.
e Task 2 involved participants finding trials to satisfy criteria for a hypothetical patient, so it did not make sense to ask questions related to their own participation.
Study results for low health literacy participants.
Measure
Task 1
Task 2
Agent
(n=15)
Control
(n=8)
P
Agent
(n=15)
Control
(n=8)
P
Completed task, n (%)a
12 (80)
8 (100)
.18
12 (80)
8 (100)
.18
Declared found a trial, n (%)a
6/14 (43)
1/8 (13)
.14
5/14 (36)
1/8 (13)
.24
Found a correct trial, n (%)a
—b
—b
—b
5/14 (36)
0/8 (0)
.05
Elapsed time (minutes), mean (SD)c
13.3 (11.4)
8.2 (6.3)
.47
6.8 (4.3)
4.6 (4.3)
.25
Number of trials examined (self-report), mean (SD)c
3.0 (3.4)
0.9 (1.1)
.16
2.8 (3.2)
1.5 (2.1)
.36
Trials examined that meet criteria (self-report; % of trials examined), mean (SD)c,d
74 (30)
75 (35)
.89
80 (31)
67 (58)
.92
To what degree did the trial match what you were looking for? (range 1-7), mean (SD)c
4.1 (1.9)
2.4 (1.4)
.06
5.3 (2.1)
3.3 (2.0)
.04
How likely are you to sign up for the trial that you found? (range 1-7), mean (SD)c
3.8 (1.1)
3.6 (1.9)
.81
—e
—e
—e
How much time do you feel it took to use the system? (range 1-7), mean (SD)c
3.8 (0.9)
4.1 (2.5)
.83
3.9 (0.3)
4.5 (2.3)
.86
How satisfied were you with the clinical trial search system? (range 1-7), mean (SD)c
5.3 (1.6)
2.9 (1.7)
.01
5.7 (1.6)
2.9 (1.4)
.002
How frustrated do you feel right now? (range 1-7), mean (SD)c
2.9 (2.0)
4.8 (2.1)
.01
2.3 (2.1)
3.1 (1.9)
.15
How pleased do you feel right now? (range 1-7), mean (SD)c
5.5 (1.6)
3.3 (2.7)
.04
5.7 (1.7)
2.6 (1.1)
.001
How much pressure did you feel to volunteer for a trial? (range 1-7), mean (SD)c
1.1 (0.3)
2.3 (2.2)
.10
—e
—e
—e
How much information do you feel was presented by the system? (range1-7), mean (SD)c
4.8 (1.5)
4.3 (1.5)
.21
—e
—e
—e
How likely would you be to use the system again, if you wanted to find another trial? (range 1-7), mean (SD)c
5.5 (1.8)
5.0 (2.4)
.74
—e
—e
—e
How likely would you be to recommend the system to someone else who was looking for a trial? (range 1-7), mean (SD)c
6.1 (1.9)
5.1 (2.7)
.45
—e
—e
—e
How much do you trust the information you received from the system? (range 1-7), mean (SD)c
6.3 (1.0)
5.1 (2.1)
.71
—e
—e
—e
a Chi-square test.
b Task 1 involved participants finding trials they were interested in, so there was no way to objectively assess whether the trials they found were “correct”.
c Mann-Whitney U test.
d Trials examined that meet criteria was a subjective self-report measure.
e Task 2 involved participants finding trials to satisfy criteria for a hypothetical patient, so it did not make sense to ask questions related to their own participation.
Analyzing differences between low and high health literacy participants across all study conditions indicated a few significant differences. Participants with low health literacy were more likely overall to state that the trials they read satisfied their criteria compared to those with high health literacy (task 1: 74% vs 37%, U=82, P<.001; task 2: 77% vs 49%, U=148, P=.02). Participants with low health literacy were more likely to say they would sign up for the trial they found in task 1 (rating mean 3.8, SD 1.4 vs mean 2.8, SD 1.8, U=380, P=.048) and that they would recommend the system to a friend (rating mean 5.7, SD 2.3 vs mean 4.5, SD 2.3, U=379, P=.01) compared to those with adequate health literacy.
DiscussionPrincipal Results
In our comparison of a conversational agent-based search user interface to a conventional keyword- and facet-based search engine interface, participants were more satisfied with the agent and felt the agent was better at finding trials that matched their criteria compared to the conventional interface. Participants also felt more pleased and less frustrated after interacting with the agent compared to the conventional interface.
In our standardized task (task 2), it is notable that none of the low health literacy participants were able to find a correct clinical trial using the conventional search engine interface, whereas 36% (5/14) were able to do so with the conversational agent. These results reinforce our earlier findings that conventional search interfaces are unusable by individuals with low health or computer literacy [9]. It is encouraging that the conversational interface was able to provide accessibility to at least a third of these users, while being rated more highly on satisfaction by all users, including those with high health literacy. Nonetheless, it appears that research on additional adaptations is warranted in order to succeed with an even broader portion of the population.
The conversational interface does take more time to use compared to the conventional interface: 40% longer in task 1 and 27% longer in task 2 (although these differences were not statistically significant). There are several reasons for this: the time required to hear spoken prompts rather than reading them, the interview by the agent to obtain search criteria, and social dialog, tutorials, and other “off-task talk” used by the agent to improve approachability, engagement, and comprehension. However, users in our target demographic are clearly happy to spend the extra time with the conversational user interface to obtain better results; in the browsing task, they chose to spend almost twice as long finding a trial compared to those using the conventional interface. In addition, their subjective impression of time taken in task 2 indicates that users felt the conversational agent interface actually took significantly less time to use compared to the conventional interface.
Limitations
Our study had several limitations beyond the small number of participants involved. Some (21%) of the users in our study had previously been involved in clinical trials and thus are not representative of the general population of people with cancer because they may have had higher than average background knowledge about clinical trials. However, people with prior trial experience were randomly assigned to the 2 trial arms, so both groups should be equivalent in this regard. Another limitation relates to the use of the REALM as a measure of health literacy. Although this tool successfully differentiated among participants regarding the impact of our intervention, other measures could potentially have provided a more refined capacity to delve deeper within specific dimensions of electronic and computer literacy. Although the eHealth Literacy Scale (eHEALS) was designed for such a purpose, it is a subjective self-report measure [35]. We opted for an objectively scored measure in the current study; future research is warranted to further differentiate how a conversational search interface may ameliorate various dimensions of low health and computer literacy.
Comparison With Prior Work
Several studies have investigated the use of standard keyword-based search interfaces for users with low domain knowledge [14], who speak a foreign language [15], who are children [16] or older adults [17], which all share characteristics with our task and population. These studies have demonstrated that even the simplest keyword-based search interfaces are unusable for many users and that special design considerations—such as simplifying results [17] and providing language and interaction support [16]—are important for users, especially those with low health or computer literacy
Other studies have investigated the use of conversational agents to communicate health information to individuals with low health literacy. Bickmore et al [20,36] and Wang et al [37] have developed conversational agents for physical activity promotion, hospital discharge instruction, explanation of medical documents, and family health history-taking to individuals with low health literacy. Most of these studies have demonstrated that participants with low health literacy have significantly higher levels of satisfaction with conversational interfaces compared to participants with adequate health literacy.
Conclusions
Our findings suggest that conversational agent-based search engine interfaces could be a good alternative to conventional Web form-based interfaces for many kinds of applications, but especially for those intended for low health literacy users or those with limited computer experience or skills.
AbbreviationsNCI
National Cancer Institute
REALM
Rapid Estimation of Adult Literacy in Medicine
Thanks to Barbara Barry, Ramesh Manuvinakurike, Juan Fernandez, Lazlo Ring, and Maryam Aziz with their help developing the system. This work was funded by NIH National Cancer Institute grant R01CA158219.
None declared.
National Cancer Institute2015-12-23Health Information National Trends Surveyhttp://hints.cancer.gov/question-details.aspx?dataset=41&method=mail&qid=7576cENkpafCFoxSDugganMPew Research Center201301152015-12-23Health Online 2013http://www.pewinternet.org/2013/01/15/health-online-2013/6e0HETiK0AgreeEMKingACCastroCMWileyABorzekowskiDL"It's got to be on this page": age and cognitive style in a study of online health information seekingJ Med Internet Res2015173e7910.2196/jmir.335225831483v17i3e79PMC4389106van DijkJDigital divide research, achievements and shortcomingsPoetics2006344-5221235Nielsen-BohlmanLPanzerAKindigDHealth Literacy: A Prescription to End Confusion20042015-12-23Washington, DCNational Academies Presshttp://www.ncbi.nlm.nih.gov/books/NBK216032/6e0HlhGIIDivianiNvan den PutteBGianiSvan WeertJCLow health literacy and evaluation of online health information: a systematic review of the literatureJ Med Internet Res2015175e11210.2196/jmir.401825953147v17i5e112PMC4468598AtkinsonNLSapersteinSLMassettHALeonardCRGramaLManrowRUsing the Internet to search for cancer clinical trials: a comparative audit of clinical trial search toolsContemp Clin Trials20080729455556410.1016/j.cct.2008.01.00718346942S1551-7144(08)00008-6PMC2724745National Cancer Institute20142015-11-29Find NCI-supported clinical trialshttp://www.cancer.gov/about-cancer/treatment/clinical-trials/search6dO9qQfUXUtamiDBickmoreTWBarryBPaasche-OrlowMKHealth literacy and usability of clinical trial search enginesJ Health Commun201419 Suppl 219020410.1080/10810730.2014.93884225315593Paasche-OrlowMKWolfMSThe causal pathways linking health literacy to health outcomesAm J Health Behav200731 Suppl 1S19S2610.5555/ajhb.2007.31.supp.S1917931132Paasche-OrlowMKParkerRMGazmararianJANielsen-BohlmanLTRuddRRThe prevalence of limited health literacyJ Gen Intern Med20050220217518410.1111/j.1525-1497.2005.40245.x15836552JGI40245PMC1490053VolandesAEPaasche-OrlowMGillickMRCookEFShaykevichSAbboEDLehmannLHealth literacy not race predicts end-of-life care preferencesJ Palliat Med20080611575476210.1089/jpm.2007.022418588408HearstMSearch User Interfaces2009CambridgeCambridge University PressZhangXLiuJYuanXColeMBelkinNLiuCDo users with different domain knowledge select different sets of documents?Proceedings of the Annual Symposium on Human-Computer Interaction and Information Retrieval (HCIR)2012Symposium on Human-Computer Interaction and Information Retrieval (HCIR)Oct 4-5 2012Cambridge, MAKomlodiAJozsaEKolesMHercegfiKSearch quality differences in native and foreign language searching2011Annual Symposium on Human-Computer Interaction and Information Retrieval (HCIR)Oct 20, 2011Mountain View, CAGossenTNitscheMNurnbergerAKnowledge journey: a Web search interface for young users2012Annual Symposium on Human-Computer Interaction and Information Retrieval (HCIR)Oct 4-5, 2012Cambridge, MA10.1145/2391224.2391225AulaAKäkiMLess is more in Web search interfaces for older adultsFirst Monday2005105100101AllamASchulzPJNakamotoKThe impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: two experiments manipulating Google outputJ Med Internet Res2014164e10010.2196/jmir.264224694866v16i4e100PMC4004139MackertMKahlorLTylerDGustafsonJDesigning e-health interventions for low-health-literate culturally diverse parents: addressing the obesity epidemicTelemed J E Health20090915767267710.1089/tmj.2009.001219694596PMC2956510BickmoreTWPfeiferLMByronDForsytheSHenaultLEJackBWSillimanRPaasche-OrlowMKUsability of conversational agents by patients with inadequate health literacy: evidence from two clinical trialsJ Health Commun201015 Suppl 219721010.1080/10810730.2010.49999120845204926954323BolNvan WeertJCde HaesHCLoosEFSmetsEMThe effect of modality and narration style on recall of online health information: results from a Web-based experimentJ Med Internet Res2015174e10410.2196/jmir.416425910416v17i4e104PMC4425819MeppelinkCSvan WeertJCHavenCJSmitEGThe effectiveness of health animations in audiences with different health literacy levels: an experimental studyJ Med Internet Res2015171e1110.2196/jmir.397925586711v17i1e11PMC4319081CassellJEmbodied Conversational Agents2000Cambridge, MAMIT PressBickmoreTGruberAPicardREstablishing the computer-patient working alliance in automated health behavior change interventionsPatient Educ Couns200510591213010.1016/j.pec.2004.09.00816198215S0738-3991(04)00307-6ReiterEDaleRBuilding Natural Language Generation Systems2000CambridgeCambridge University PressBickmoreTSchulmanDShawGDTask and LiteBody: open source, standards-based tools for building Web-deployed embodied conversational agentsIntelligent Virtual Agents20099th International Conference, IVA 2009Sep 14-16, 2009Amsterdam42543110.1007/978-3-642-04380-2_46BickmoreTCarusoLClough-GorrKHeerenT"It's just like you talk to a friend"-relational agents for older adultsInteract Comput2005176711735CorbinJStraussABasics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory2014Thousand Oaks, CASageQuinlanJInduction of decision treesMach Learn19861181106DavisTCLongSWJacksonRHMayeauxEJGeorgeRBMurphyPWCrouchMARapid estimate of adult literacy in medicine: a shortened screening instrumentFam Med1993062563913958349060LindauSTBasuALeitschSAHealth literacy as a predictor of follow-up after an abnormal Pap smear: a prospective studyJ Gen Intern Med20060821882983410.1111/j.1525-1497.2006.00534.x16881942JGI534PMC1831576MancusoCARinconMImpact of health literacy on longitudinal asthma outcomesJ Gen Intern Med20060821881381710.1111/j.1525-1497.2006.00528.x16881939JGI528PMC1831574SudoreRLYaffeKSatterfieldSHarrisTBMehtaKMSimonsickEMNewmanABRosanoCRooksRRubinSMAyonayonHNSchillingerDLimited literacy and mortality in the elderly: the health, aging, and body composition studyJ Gen Intern Med20060821880681210.1111/j.1525-1497.2006.00539.x16881938JGI539PMC1831586LincolnAPaasche-OrlowMKChengDMLloyd-TravagliniCCarusoCSaitzRSametJHImpact of health literacy on depressive symptoms and mental health-related: quality of life among adults with addictionJ Gen Intern Med20060821881882210.1111/j.1525-1497.2006.00533.x16881940JGI533PMC1831585GhaddarSFValerioMAGarciaCMHansenLAdolescent health literacy: the importance of credible sources for online health informationJ Sch Health201201821283610.1111/j.1746-1561.2011.00664.x22142172BickmoreTWPfeiferLMPaasche-OrlowMKUsing computer agents to explain medical documents to patients with low health literacyPatient Educ Couns20090675331532010.1016/j.pec.2009.02.00719297116S0738-3991(09)00046-9PMC2692364WangCBickmoreTBowenDJNorkunasTCampionMCabralHWinterMPaasche-OrlowMAcceptability and feasibility of a virtual counselor (VICKY) to collect family health historiesGenet Med201510171082283010.1038/gim.2014.19825590980gim2014198PMC4503525