The ever-growing amount of health information available on the web is increasing the demand for tools that provide personalized and actionable health information. In addition, patients avidly seek information to inform their own health care decisions, either directly or by verifying information discussed during professional consultations. The broad scope of web-based health information includes generic information obtained through web-based searches and decision aids and tools that deliver personalized advice based on information specific to users. Such tools include symptom checkers that provide users with a potential diagnosis after responding to a set of probes about their symptoms.

Web-based symptom checkers are becoming increasingly popular, and the emergence of the COVID-19 pandemic has increased interest in these tools [1]. However, only a few studies have examined how and why they are used [2-4]. The limited research on symptom checkers has found generally positive effects of their use; technologically sophisticated web-based triage systems may help reduce unnecessary visits to emergency rooms and overuse of antibiotics [4], make health care accessible in low-resource settings [5], and increase patient engagement [6]. However, although the potential for their utility is great, more research is needed on the actual use and effects of such tools.

Some studies have raised concerns about the potential of web-based health information systems to spread disinformation and inaccurate diagnostic information [2,7,8]. For example, a study evaluating the diagnostic and triage accuracy of 23 web-based symptom checkers found that physicians performed better than the symptom checker algorithms [4]. However, physicians made incorrect diagnoses in 15% of the cases. Although research suggests that symptom checkers may be less effective than physicians in terms of diagnostic accuracy, it might be more critical that symptom checkers provide recommended actions (eg, whether symptoms warrant a trip to the hospital). Therefore, it is important to understand the impact of symptom checkers on how patients seek care and respond to health care advice.

A significant potential contribution of web-based symptom checkers as triage systems may be to reduce the negative effects of the current overwhelming health information environment, such as the health information overload experienced by web-based health information seekers and their struggle to discern reliable information from misinformation. A web-based medical information system that addresses the abovementioned problems can help people better understand the potential causes of the symptoms they are experiencing, empower them to seek the right kind of help, and potentially reduce anxiety caused by the symptoms they are experiencing.

Users must be able to trust and follow their recommendations for web-based symptom checkers to make meaningful contributions. If web-based symptom checkers are not trusted, they are less likely to be adopted by users, thereby limiting their potential [9]. Moreover, users may be unclear about the technology behind web-based symptom checkers. Research suggests that web-based symptom checkers’ artificial intelligence (AI) systems are neither transparent nor comprehensible to users, which may undermine trust in such tools [10]. Nevertheless, despite hesitancy and concerns regarding the accuracy, AI-powered symptom checkers have been perceived as useful for diagnosis by users [11].

A large body of research on information seeking grounded in the uses and gratification frameworks [12] has examined how people use different media to fulfill or gratify various needs. Research in this tradition has characterized health information–seeking behaviors by sources (ie, web-based vs offline seeking [13,14]) or objective (ie, seeking for themselves vs others [15-17]). Multiple studies have confirmed that active information seekers from nonclinical sources, including the internet, are more likely to be White, female, and have relatively high levels of education and income [18-22]. Racial differences in health information–seeking, as well as confidence in information and trust in various sources, have been well documented. There may be different levels of trust and use of sources by racial groups, which can lead to disparities if inaccurate sources are used [23].

Research based on self-reported media use has established that deliberate information seeking from media, including the internet, has been associated with better health outcomes [24], increased engagement in prevention behaviors [25], and more positive patient-clinician interactions [26-28] and has also assisted individuals in coping with uncertainty [20]. Web-based health information–seeking before presenting to an emergency physician also has the potential to improve patient-provider interaction without negatively affecting adherence to treatment [29].

Despite this extensive body of research on information seeking and the importance of the internet and other “new” media as sources of medical information, the quality of the evidence for the effects has been limited. Most previous studies examining information seeking from nonclinical sources, including nearly all internet-based health information–seeking studies, are limited by their reliance on self-reports of individuals’ information-seeking behaviors and behavioral or psychosocial outcomes. Furthermore, most studies rely on generalized, non–time-bound health information–seeking behaviors (ie, “Have you ever looked for information about [a topic] from [a source]”), or ask about information seeking within a specific timeframe, but do not examine the content of the information retrieved or the recommendation provided. Thus, the next frontier in this line of research is one that links objective measures of information seeking—both sources and content—with clinical and psychosocial outcomes to understand how people use the information they seek and find from nontraditional sources.

This study aimed to address the methodological limitations of prior information-seeking research and examine who seeks information from an intelligent web-based symptom checker and for what purpose, how users experience the tool, what they intend to do with the information, and predictors of intentions to follow the tool recommendations. The following research questions (RQs) guided this study:

This cross-sectional study used data from patient encounters using Buoy Health, an AI-powered web-based symptom checker, between January 14, 2019, and February 28, 2019. Founded in 2014 by a team of physicians and researchers, the tool is based on conversational medical interviewing, mirroring a conversation with a provider. At the time of writing, Buoy’s symptom checker remains accessible for free on the web or through an app to any internet-connected person. The AI-powered tool uses a progressive series of health questions communicated via a chatbot to assess user symptoms (Figure 1). Buoy’s triage or diagnostic system by design offers health information customized for the user.

Buoy’s proprietary algorithm sources data from >18,000 clinical research studies [31]. Users receive 3 possible diagnoses and recommendations for appropriate levels of care (Figure 2). According to Buoy, the tool’s diagnostic accuracy is 90% [32]. Thus, tools such as Buoy—and other intelligent symptom checkers—have the potential to cut through the clutter of too much and contradictory information to provide personalized, science-based recommendations. A study examining how patients’ use of Buoy affected their plans for seeking care found that Buoy decreased uncertainty among users [33]. Buoy also lowered the level of urgency in patients associated with their condition. This study suggests tools such as Buoy are associated with users’ intended behavior when seeking care based on triage questions. Accordingly, our study adds to the growing literature that seeks to understand how patients use tools such as Buoy together with their providers to manage their health.

A recruitment script was shown to Buoy users, assumed to be web-based health information seekers, who met the inclusion criteria via a pop-up window immediately following their Buoy session. Inclusion criteria included completion of the Buoy interview to the recommendation stage in ≤10 minutes, being aged ≥18 years, and residency in the United States (although not necessarily physically in the United States at the time of seeking).

In addition, potential participants must have completed the Buoy interview for themselves; that is, they were seeking information about their symptoms. As 95% of Buoy users complete the diagnostic interview within 10 minutes, users who took longer were not representative of the typical user and thus were not invited to participate in our study to avoid other unanticipated ways in which they might differ from the typical user. Using similar logic, we excluded people who had a pre-existing serious or chronic condition [34] as they may not be representative of the typical Buoy user either. It is expected that their health information–seeking habits and use patterns of Buoy would be different from all other Buoy users. Finally, for ethical reasons, we excluded users who Buoy advised to seek immediate medical care from eligibility, including immediate medical care via 911 or in the emergency department. Figure 3 shows attrition at each stage.

Participants received a US $5 electronic gift card in appreciation of their time following completion of the survey, which had a mean time to completion of 8.61 (SD 6.78) minutes. The gift cards were delivered to an email address that was also used for follow-up. Participants were informed that they would receive another incentive (US $10) following the completion of a second survey. A follow-up assessment was administered 2 weeks after the initial survey to those who chose to provide an email address; however, this study reports only the baseline data.

Users’ self-reported symptoms resulting from the Buoy interview were coded into 13 categories using the Centers for Disease Control and Prevention National Ambulatory Medical Care Survey (NAMCS) coding protocol [42]. During the interview, the users were prompted to enter up to 5 presenting symptoms. We report only on the first as that was the primary issue driving the use of the web-based symptom checker. Using an iterative coding process, we generated a set of unique symptoms (N=2040) and unique diagnoses (N=938) from all Buoy data sets.

A total of 2 coders independently coded the first symptom. Coder 1 was part of the codebook development process. Coder 2 was introduced into the study once the codebook was finalized. Disagreements were resolved by discussion. The second author resolved disagreements when consensus was not possible. Cohen κ was run to determine interrater reliability between the 2 coders’ assignment of NAMCS codes for the 2040 unique symptoms; there was substantial agreement between the 2 coders (κ=0.73; [43]). We further categorized whether the first symptoms were serious and likely to require medical attention based on Shapiro et al [44] (chest pain that is heart related, bleeding, loss of consciousness, shortness of breath, and weight loss).

Users are provided with up to 3 possible diagnoses or display names at the completion of their interview, ranked and weighted according to Buoy’s proprietary algorithm, along with recommendations for subsequent actions. Diagnoses were coded into 25 categories comprising major systems, disorders, and conditions, in line with the NAMCS. We report the first diagnosis display name as the algorithm had the highest confidence in it. In addition, the first diagnosis display name had no missing data.

For this descriptive analysis, we computed frequencies and percentages to summarize participant characteristics and experiences, overall and by sex and ethnicity where relevant, and to assess intentions to comply with Buoy recommendations. ANOVAs with Bonferroni correction examined the mean outcome differences between racial or ethnic groups on user experience and recommendations. Nonparametric tests in the form of the Wilcoxon rank sum test were performed to test the mean differences between sexes in user experience and recommendations. The relationship between perceptions of the symptom checker and intention to follow recommended actions was assessed using logistic regression. Logistic regression models examined the factors affecting confidence in recommendations and intention to follow these. Analyses were conducted using R (version 4.0.3; R Foundation for Statistical Computing).

The University of California, Merced Institutional Review Board approved this study (approval number: UCM2018-124).

Consistent with prior studies on health information seekers, Buoy users were well-educated (1384/1704, 81.22% some college or more), mostly White (1227/1693, 72.47%), and female (2069/2437, 84.89%). The mean age of the users was 39.4 (SD 14.7) years. Users were similar to other users of web-based symptom tools, and a prior study of web-based symptom checkers found that users were predominantly female with a mean age of 40 years [33]. Findings from other studies further indicate an age, sex, and socioeconomic divide among adults’ web-based health information–seeking behaviors [45]. The sampled users were also relatively privileged in terms of health care access; most had insurance (1449/1630, 89%) and a regular health care provider (1307/1709, 76%). They were generally in good health; 59% (1000/1703) reported their health as good, very good, or excellent. Table 1 shows the additional demographic details.

Users selected ≥1 of the 5 stated reasons for using Buoy, as well as open-ended responses, which were coded into 5 new categories. Over one-third (839/2437, 34.43%) of the users indicated persistent symptoms as a reason for using Buoy, followed by new symptoms (767/2437, 31.47%), symptoms not serious (545/2437, 22.36%), sensitive or embarrassing symptoms (269/2437, 11.04%), and not confident that health care provider provided correct diagnosis (220/2437, 9.03%). Less common reasons included new categories/codes: curiosity (66/2437, 2.71%), access (36/2437, 1.48%), anxiety (16/2437, 0.66%), triage (29/2437, 1.19%), and other (30/2437, 1.23%; data not shown).

We report the patterns in symptoms and diagnoses in 2 ways. First, we report the frequencies of symptoms and diagnoses organized by the NAMCS Biological Systems associated with them (Multimedia Appendix 2 [42,44]). Second, we report the top 10 symptoms and diagnosis categories, overall and by sex and race/ethnicity (Table 2).

Only 3 types of symptoms—pain (855/2437, 35.08%), gynecological issues (293/2437, 12.02%), and masses or lumps (204/2437, 8.37%)—accounted for almost half (1352/2437, 55.48%) of the site visits. The top 3 symptoms entered by men included pain (154/368, 41.8%), masses or lumps (49/368, 13.3%), and skin issues (33/368, 8.9%), whereas the top 3 symptoms in women included pain (701/2069, 33.88%), gynecological issues (293/2069, 14.16%), and masses or lumps (155/2069, 7.49%). Pain, gynecological issues, and masses or lumps were also reported as the top 3 symptoms for White, Black, and Asian or Pacific Islander users. The top 3 symptoms in Latino users were pain (43/139, 30.9%), gynecological issues (22/139, 15.8%), and gastrointestinal problems (14/139, 10.1%). In comparison, Native Americans, who represented <1% of users, only entered five symptoms: pain (4/13, 31%), gynecological issues (4/13, 31%), skin issues (1/13, 8%), gastrointestinal problems (1/13, 8%), and impaired sensation (1/13, 8%).

Among the entire sample, major diagnoses were musculoskeletal (303/2437, 12.43%), gynecological (304/2437, 12.47%) and skin conditions (297/2437, 12.19%), and infectious diseases (300/2437, 12.31%). Comparably, the top 3 diagnoses reported by Buoy for men included musculoskeletal conditions (63/368, 17.1%) and skin conditions (62/368, 16.8%) and infectious diseases (37/368, 10%). The top 3 diagnoses for women included gynecological conditions (304/2069, 14.69%), infectious diseases (263/2069, 12.7%), and musculoskeletal conditions (240/2069, 11.59%). The diagnoses based on race or ethnicity followed a similar pattern. White users also reported musculoskeletal conditions (177/1227, 14.42%), infectious diseases (163/1227, 13.28%), and skin conditions (148/1227, 12.06%) as the top 3 diagnoses. Latino, Black, and Asian or Pacific Islander users reported gynecological conditions, skin conditions, and infectious diseases as the top 3 diagnoses. Gynecological conditions were reported as the top diagnosis category by Black (42/189, 22.2%) and Asian or Pacific Islander (17/86, 20%) users, whereas Latino users reported infectious diseases (26/139, 18.7%) as the top diagnosis category.

Buoy’s primary recommendation was more evenly split across the less-serious triage categories. Users self-reported primary care physicians in 2 weeks (754/2141, 35.22%), self-treatment (452/2141, 21.11%), and primary care in 1 to 2 days (373/2141, 17.42%) as the top 3 recommendations provided by Buoy, followed by wait and watch (339/2141, 15.83%). Only 5.74% (123/2141) and 4.67% (100/2141) of users self-reported phone calls or in-person visits in the next 3 days and in-person visits that day or as soon as possible, respectively. The recommendations reported by Buoy closely matched primary care physicians in 2 weeks (924/2437, 37.91%), self-treatment (552/2437, 22.65%), and primary care in 1 to 2 days (456/2437, 18.71%). Most users (2098/2437, 86.09%) had 2 recommendations. Approximately 71.85% (1751/2437) had 3 recommendations, as reported by Buoy.

Users generally reported high levels of confidence in Buoy (mean 3.47, SD 0.97), found it useful (mean 4.18, SD 0.81) and easy to understand (mean 4.64, SD 0.53), and said that Buoy made them feel less anxious (mean 3.60, SD 1.05) and more empowered to seek medical help (mean 3.75, SD 0.96) Compared with White users, Latino and Black users had significantly more confidence in Buoy (P<.05), and the former also found it significantly more useful (P<.05; Table 3). Consistent with prior studies on trust in web-based health information sources [46-48], physicians were the most trusted source. However, Buoy was trusted more (mean 3.68, SD 0.61) than any other nonmedical source, including government agencies (mean 2.85, SD 0.95), family (mean 2.64, SD 0.76), and Google (mean 2.52, SD 0.79).

Overall, most users reported intentions to follow Buoy’s recommendations (1428/1886, 75.71%) and discuss Buoy’s recommendations with a physician or health care professional (1198/1830, 65.44%; Table 4). Users reported the strongest intention to follow Buoy’s wait and watch recommendation (mean 4.38, SD 0.90), followed by self-treatment (mean 4.33, SD 0.93), in-person visit that day or as soon as possible (mean 4.17, SD 1.01), phone call or in-person visit in the next 3 days (mean 4.05, SD 1.05), primary care physician in 2 weeks (mean 3.92, SD 1.19), and primary care physician in 1 to 2 days (mean 3.68, SD 1.26).

Intention to discuss Buoy’s recommendations was positively associated with having a regular provider (odds ratio [OR] 1.37, 95% CI 1.04-1.82), and an income >US $50,000 was negatively associated (OR 0.75, 95% CI 0.57-0.98; OR 66, 95% CI 0.48-0.91; Table 5). Users aged between 35 and 44 years (OR 1.51, 95% CI 1.13-2.03) and 45 and 64 years (OR 1.57, 95% CI 1.18-2.10) had better intentions of discussing recommendations than younger users (aged 18-34 years). Compared with White users, Latino (OR 1.96, 95% CI 1.22-3.25) and Black (OR 2.37, 95% CI 1.57-3.66) users had stronger intentions to discuss recommendations with a provider, and Black users were twice as likely to intend to do so. Confidence in Buoy (OR 1.54, 95% CI 1.34-1.76), perceived utility (OR 1.32, 95% CI 1.10-1.58), and anxiety reduction because of using Buoy (OR 1.43, 95% CI 1.24-1.63) were associated with higher intention to discuss Buoy’s recommendations.

Overall, users had strong intentions to follow Buoy’s recommendations, and users who self-reported very good or excellent health had the strongest intention to wait or watch or self-treat (OR 1.92, 95% CI 1.04-3.65; Table 5). Those who reported Buoy as easy to read and understand were 2.2 times (95% CI 1.21-4.14) more likely to intend to wait or watch or self-treat than those who reported lower comprehensibility for Buoy. Users with health insurance (OR 2.21, 95% CI 1.36-3.57) and a regular provider (OR 1.59, 95% CI 1.11-2.28) had the strongest intentions to seek care. Confidence in Buoy (OR 1.87, 95% CI 1.56-2.25) and anxiety reduction because of Buoy (OR 1.54, 95% CI 1.29-1.83) were also associated with a higher intention to seek care.

This study sought to understand who uses web-based AI-powered symptom checkers and for what purposes. The demographic profile of Buoy users was similar to that described in other studies of web-based health information seekers, suggesting that older, marginalized groups continue to be digitally excluded. Consistent with data on internet-based health-seeking behaviors more generally [49], most Buoy users were middle-aged (or younger), female, and highly educated. More research is needed to better understand older adults’ web-based health information–seeking behaviors and support their medical and health decisions [50]. Although a scoping review of articles examining AI-driven symptom checkers from various perspectives found that those who do not have access to health care services are more likely to use symptom checkers [51], Buoy users overwhelmingly reported having health insurance. This finding does not negate the possibility that users were motivated by financial considerations, as most contemporary health plans require an out-of-pocket copayment. Nevertheless, this suggests that other considerations such as convenience were also salient.

Along these lines, prior research has identified an association between stigmatizing conditions and the use of symptom checkers [51]. In this study, gynecological problems were among the top 3 symptom groups. Furthermore, across presenting symptoms or diagnoses, approximately 11.04% (269/2437) of the respondents were “too embarrassed” to seek in-person care. Taken together, these findings suggest that symptom checkers might be particularly useful for users affected by conditions considered personal, embarrassing, stigmatizing, not warranting the physician’s attention, or requiring potentially uncomfortable or psychologically stressful physical examinations (such as pelvic examinations).

In examining the reasons for using the tool, approximately one-third of the respondents had persistent symptoms that failed to resolve spontaneously, another one-third had new symptoms, and the rest either thought they did not need professional attention or (as mentioned previously) were too embarrassed to seek care. Thus, some patients used the symptom checker because they had significant health-related concerns; some because they lacked sufficient concern to warrant in-person care; and some because they had issues with perceived quality, cost, or convenience of available care or simply wanted a second opinion. Symptoms that persist longer than expected have been identified as strong drivers of health-related anxiety and, thus, health care use [52]. At the same time, valuing convenience and lack of trust in the health care system (factors that may be particularly prominent among young people and racially and ethnically minoritized groups, respectively) have been associated with a lower propensity to use formal health care services [53].

Regarding the user experience, users had high levels of confidence in Buoy and found it useful. Moreover, users trusted Buoy more than any other nonmedical source. Perceived confidence, utility, and trust were associated with a stronger intention to discuss Buoy’s recommendations with a physician. This finding is in line with a study examining patient perspectives on the usefulness of a symptom checker [11]. Most Buoy users found the tool useful for diagnosis, and most reported that they would use it again. Although the experiences of users who discussed recommendations with their physicians varied, most felt that physicians were open to discussing the results of the tool. This is an important finding, as users may not follow recommendations to seek care if they believe that acting on the advice of a symptom checker will be questioned or even belittled by their physician, regardless of their confidence in the tool.

This study ultimately advances the understanding of web-based health information–seeking behaviors and outcomes by linking objective measures of information seeking from a web-based AI-powered system with clinical and psychosocial outcomes. The results demonstrate the potential utility of an artificially powered web-based health information tool to improve outcomes for users. Symptom checkers have been described as a means of addressing the lack of access to physicians and reducing unnecessary office visits [4].

There is a lack of research on whether the use of symptom checkers translates into medical care–seeking behaviors [4,33]. Future research should examine the effects of such tools on medical care seeking, specifically how users interpret recommendations, whether the recommendations are followed, and how user responses vary among sociodemographic groups. For example, one might surmise that individuals with limited access to care or with prior negative health care experiences might be more likely to attend to, appreciate, and follow such recommendations than their more privileged counterparts. Although symptom checkers may empower users to make more informed decisions, they might paradoxically worsen health disparities if their use were less accessible to some groups. Currently, web-based symptom checkers are mostly available for free. As web-based symptom checker companies establish partnerships with employers and health insurance companies to ensure profits, not all users may be equally ready or able to pay for symptom checking.

We partnered with the Buoy technical and medical staff to sample the users. Owing to our partnership approach, we were able to obtain the specific symptoms reported by the participants as the primary reason for using Buoy, as well as the possible diagnoses identified by Buoy and Buoy’s triage recommendation. This allowed for the comparison and validation of self-reported data. We also obtained from Buoy the symptoms, diagnoses, triage, and sex of eligible users who opted not to participate in our study. This allowed us to compare our sample to the population of users and assess potential bias. In addition, a benefit of a collaborative approach is the potential to overcome the self-report limitations of prior studies. Thus, we obtained from Buoy the paths that individuals took and Buoy’s final recommendation. We were also able to match the initial reason for the consultation to the reason reported in the survey and assess the extent to which respondents understood the recommendation and intended to act upon it. By leveraging a public or private partnership, we were able to explore the use and effects of a web-based symptom checker, which has important implications for health equity and the health care system during and after the COVID-19 crisis.

The limitations of this study include the use of cross-sectional data, which limited the ability to make any causal inferences, and the potential lack of applicability to other web-based symptom checkers. In addition, we did not assess the actual search terms entered by users. Finally, our study used a limited definition of web-based health information. Searches for symptoms using a web-based symptom checker differ from other forms of health-related information communicated through the internet. For example, web-based health communities can also be a source of social support [54] and peer-to-peer medical advice [7].

The results of this study demonstrate the potential utility of a web-based health information tool to empower people to seek appropriate care and reduce health-related anxiety. An interactive symptom checker might provide more personalized and potentially reliable medical information than other forms of web-based health information–seeking. Despite encouraging results suggesting that the web-based tool may fulfill unmet health information needs among women and Black and Latino adults, analyses of the user base illustrate persistent second-level digital divide effects.

For web-based symptom checkers to make a meaningful contribution, they must not only be trusted by users but also meet their diverse needs, especially those concerning usability and comprehensibility. The inability to access web-based symptom checkers may also be associated with increased disparities in access to care, particularly among groups that have lagged historically in terms of digital access and literacy. Moreover, web-based symptom checker business models may further exacerbate these disparities. In contrast, AI technologies such as Buoy have the potential to alleviate disparities by allowing users to access accurate, actionable, and personalized advice within an evolving but often confusing web-based health information environment. Finally, there is a lack of evidence on whether web-based symptom checkers influence care-seeking behaviors. To address this gap, future research will use Buoy users’ follow-up data to assess the extent to which users discuss their web-based findings with physicians, as well as barriers to the same and patient satisfaction.