Studies that investigate triage errors in primary care are scarce. A systematic review assessing the safety of telephone triage in out-of-hours care compared with standard face-to-face doctor assessment suggested that triage was safe in 97% (95% CI 96.5%-97.4%) of all patients contacting out-of-hours care and in 89% (95% CI 86.7%-90.2%) of patients with high urgency [11]. This reduced to 46% (95% CI 42.7%-49.8%) when high-risk groups were examined [11]. A triage system in Belgium reported a comparable level of accuracy (98%) when a new French-language algorithm was used [12]. This seems to be consistent with reported rates of triage errors since the 1970s [13]. However, a more recent study in Belgium that compared the triage decisions made by telephone operators and those made by physicians showed a lower level of accuracy [14]. The correctness of the advice given by the operator according to the physicians was 71%, with 12% underestimation of urgency and 17% overestimation [14].

Although some primary care telephone triage is done by doctors, much is done by nurses, sometimes using computer-based clinical decision support systems [15]. A study assessing the safety of telephone triage in general practitioner cooperatives found that triage nurses estimated the level of urgency correctly in 69% of total patients and underestimated the level of urgency in 19% of them [16]. A similar study in the Netherlands reported a comparable rate of triage errors (ie, the level of care was underestimated in 17% of the patients and overestimated in 19%). In Belgium, both the undertriage and overtriage rates were slightly lower, at 10% and 13% of all patients who contacted the out-of-hour telephone service, respectively [17]. In the same study, general practitioners and nurses were found to agree on the level of urgency in 77% of all contacts [17].

In emergency department settings, triage error rates appear to be markedly higher. Tam et al [18] found that triage accuracy in a number of multicentered and single-centered studies was only around 60%, with about 23% of cases undertriaged. A similar rate of triage errors was indicated in a US study, where emergency nurse triage accuracy was recorded for 54% of patients with acute myocardial infarction [19]. Better triage accuracy was recorded in a study in South Korea, where retrospective comparison of records of patients admitted to two emergency departments with a gold standard method (based on a 5-level triage scale reviewed by medical experts) [20] found disagreement in 14.7% of the cases (10% overtriage and 5% undertriage). A comparable 17% triage error rate was reported in a study in Brazil using similar methods [21]. Although triage accuracy varied across studies and there is no standardized acceptable triage rate for all patients, the American College of Surgeons has suggested an acceptable rate of undertriage for trauma patients of 5% and 25%-35% for overtriage [18,22]. It is worth noting that relatively high overtriage rates may be seen in emergency care settings where access to rapid imaging or other investigations allows for subsequent “downgrading” of triage.

The accuracy of digital symptom checkers in providing triage has been met with skepticism. There is limited evidence in this area, but vignette studies have suggested that triage error rates have been shown to be high for digital symptom checkers [23,24]. One study compared 12 publicly available symptom checkers and reported that only 51% of triage decisions for the top 5 diagnoses were correct [23]. However, this is the mean rate of errors, which may be skewed by a wide range of triage accuracy between the least and most accurate symptom checkers (22%-72%) [23]. The rates of triage errors increase with condition urgency [23,25]. The level of urgency was found to be appropriately assessed in a small proportion of emergency cases with ophthalmic diagnoses (39%, 95% CI 14%-64%) [26]. When applied in emergency department settings, symptom checkers were reported to be inadequately sensitive to emergency cases, with triage accuracy between 45%-75% of total patients [27-30]. However, in a recent study using digital patient self-triage in a hospital emergency department, a digital tool showed higher sensitivity to high-acuity conditions and similar specificity for low-acuity conditions when compared with standard nurse triage using the Manchester Triage System; it also tended to result in overtriage of patients when compared with standard nurse triage [31].

Triage advice provided by symptom checkers is found to be more risk averse than that provided by health care professionals [30,32], with 85% of the users advised to see their doctor in one study [33]. However, in a 5-year follow-up evaluation study, it was observed that symptom checkers in 2020 are less risk averse (odds of 1.11:1, overtriage errors to undertriage errors) than in 2015 (odds of 2.82:1) [24]. Triage errors in emergencies, nevertheless, are still high, with 40% of emergency cases being missed by symptom checkers [24].

Although most studies regarding the accuracy of symptom checkers were carried out through clinical vignettes [23,24], some clinical trials have been conducted to compare the rates of triage error of face-to-face consultation with a physician and digital symptom assessment technologies [34-37]. Results from these clinical trials show that while symptom checkers did not perform as well as face-to-face consultation, correct triage for certain health conditions was still achieved in a higher proportion of patients than expected [37]. Some symptom checkers were reported to attain a sensitivity level of over 50% [36], consistent with previous findings [23].

Evidence of triage error rates in primary care, in emergency care, and by symptom checkers is summarized in Table 1.

Triage error rates in primary and emergency care vary widely across the literature [38], and differing settings and definitions of triage across settings make comparison difficult. The overall level of accuracy of out-of-hour telephone triage was between 69% and 98%. Undertriage rates ranged from 10%-19% in primary care setting and 5%-23% in emergency setting. Overtriage rates ranged from 13%-19% in primary care setting and 10%-35% in emergency setting. Based on limited evidence, digital symptom checkers have relatively low triage accuracy, with a mean error rate of around 50% [25,30]. However, this is likely skewed by outliers caused by the most and least accurate tools, ranging from 22%-72% [23]. Although the errors tend to be over- rather than undertriage, with users advised to visit a doctor in 85% of cases in one study even when symptoms were appropriate for self-care [33], symptom checkers are increasingly less risk averse [24].

It is worth noting that this article is not a formal systematic review, thus no specific strategies or selection criteria were applied to our literature search. This might result in potentially relevant studies being missed, despite our best effort to ensure appropriate studies regarding triage accuracy were included. However, from our consideration of the literature, we observed a high level of heterogeneity among the rates of triage errors across studies. The heterogeneity of triage error rates in primary care, pre–primary care, and emergency care is attributable to a number of factors. Most importantly, case mix and approach to/purpose of triage differ substantially across these settings. The number and type of conditions considered in each study also differed. Although the majority of studies included a mix of acute and chronic conditions, some only considered one type of disease (eg, chronic mental health disorders). Studies that assessed triage accuracy in more conditions were more likely to report higher error rates. In addition, the methods used to identify triage errors were heterogeneous. Eight methods were commonly employed in assessing triage accuracy, namely autopsies, patient and provider surveys, standardized patients, second reviews, diagnostic testing audit, malpractice claims, case reviews, and voluntary reports [3]. Studies that used different methods were found to report significantly different rates of errors [3]. Finally, there appeared to be a lack of clarity in the definition and comparison of triage errors. Some studies did not specify whether the triage errors were overtriage or undertriage. This lack of clarity and consistency means it is not possible to draw conclusions or make clear recommendations as to an acceptable error rate for symptom checkers.

Consideration of triage error in primary care is particularly timely in the current unprecedented public health context. The recent COVID-19 pandemic has challenged the ability of health systems worldwide to meet demand, with services in some countries completely overwhelmed. A pressing need to avoid all but the most urgent and essential health service use, and to limit face-to-face interaction between health care professionals and members of the public to an absolute minimum, has led to the adoption of a “remote total triage” system in primary care using telephone and online consulting in many countries [39].

Whether the digital symptom checkers’ level of performance for triage is acceptable depends on the purposes for which they are used [25]. If symptom checkers are seen as a replacement for seeing physicians, they would currently be an inferior alternative [25]. However, if used by individuals to gather quick and accessible information about particular conditions, they are likely to be superior to self-directed internet searches using online search engines [25]. This is especially appropriate when only the best-performing symptom checkers with low triage error rates are used. It is also worth noting that artificial intelligence technology is constantly improving, potentially making it possible for triage made by digital symptom checkers to become more accurate and thus become a safe and useful addition to traditional face-to-face consultations.

Although seeking to avoid unnecessary burden on health services, the lack of available background information and inability to include information from physical examinations or nonverbal cues means that any remote assessment system will likely need to take a risk averse approach to triage. Thus, it is arguably appropriate that digital triage tools adopt this approach.

Most studies assessing the triage error rates among symptom checkers are conducted through clinical vignettes. The preparation and evaluation of vignettes need to be standardized to allow for external validity and comparability. Furthermore, clinical trials where symptom checkers’ rates of triage error are compared with those of face-to-face consultation should be encouraged. This method not only enables the assessment of triage accuracy but also allows the examination of users’ compliance with triage advice and possible benefits for the health care system.

There is little evidence on users’ compliance with triage advice, in either traditional forms of triage or that given by digital symptom checkers, nor is there data on consequences of symptom checker errors. Additionally, little is currently known about patient expectations and health beliefs in relation to digital diagnostic and triage tools. It seems likely that most individuals would place lower weighting on the advice of a symptom checker than a human clinician, and use the information provided by these tools as part of their decision-making process. However, in times of increasing reliance on digital technology, it is possible that some individuals may have greater trust in these tools than might be expected. Research is clearly needed to clarify these questions, but developers should assume a relatively high degree of reliance of users on the recommendations that symptom checkers provide.

In addition, well-conducted research to understand the clinical effectiveness of digital symptom checkers in effectively triaging individuals (ie, offering appropriate self-care advice or assigning to appropriate services) is urgently needed. To inform decisions of users and policy makers adequately, this must incorporate formal comparison with existing provision, with a focus on primary care telephone and online triage.

There is very limited evidence and no clear gold standard comparison for triage errors in digital symptom checkers, meaning that it is not possible to make recommendations on an acceptable error rate. Positioning symptom checkers in the self-care/pre–primary care triage setting therefore seems to be most appropriate and where they can likely add value for individuals experiencing symptoms. Industry and academics should work together to develop the necessary evidence, and efforts should be made to collect user feedback and outcomes data. Until clearer comparisons with existing care are available, digital symptom checkers and triage tools should appropriately continue to take a risk averse approach in the recommendations they give to users.