Sepsis is a major health concern causing significant mortality worldwide [1,2]. Although used to describe a variety of clinical conditions, the most recent recommendations suggest that sepsis should be defined as a “life threatening organ dysfunction caused by a dysregulated host response to infection” [3]. In the United States, the average annual age adjusted incidence is 300-1000 cases per 100,000 people [4]. The mortality from this condition is as high as one in four patients [1,2,5,6]. In addition, it poses a significant financial burden on health care systems. The annual total costs of sepsis in the United States was US $20 billion in 2011 [7]. Additionally, there are numerous indirect costs that may significantly impact patients’ quality of life. For example, the older population may have severe long-term health problems after sepsis, including both cognitive dysfunction and functional disability [8].

Recognizing sepsis early and initiating timely treatment results in improved patient outcomes and significant cost reduction [9,10]. The International Surviving Sepsis Campaign recommends early identification of patients with sepsis and immediate treatment such as antibiotics within 1 hour of suspecting sepsis and septic shock [11]. A retrospective multicenter study found for every hour’s delay in the treatment of patients with septic shock, the risk of death increases by 7.6% [12]. Delays in identification of sepsis are often attributable to a lack of diagnostic tools, gradual disease progression, and no gold standard for diagnosis [13]. Interventions such as regular monitoring of vital signs and elevated lactate levels aid early recognition [14]. However, despite these measures, patients are still being diagnosed late. Although track and trigger scoring systems such as the National Early Warning Score have standardized the documentation of vital signs and enabled earlier recognition to a degree, they are still subject to inaccuracies in recordings because of their subjective nature and intermittent monitoring [15]. Alternate strategies in sepsis detection are therefore urgently required to improve outcomes.

Digital technology holds significant promise in enabling early sepsis recognition. Coupled with the rapid expansion of electronic health records (EHRs) worldwide, it is now possible to provide health care staff with real-time information on laboratory tests, imaging, and physiological vital signs at the bedside. This is particularly relevant in sepsis because of the traditional reliance on risk scoring systems used to define sepsis. These include the Systemic Inflammatory Response Syndrome (SIRS) and Sequential Organ Failure Assessment (SOFA) score [16,17]. Although usually calculated through paper-based observational charts alongside separate interfaces to retrieve blood test results, both have the potential to be automated using EHRs to reduce error and enhance digital alerting. There is now increased literature in this field that can support new clinical decisions in sepsis management. The aim of our paper was to offer an up-to-date systematic review of digital alerting systems on patient sepsis outcomes.

We applied the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for this review [18]. Our search on February 12, 2019, included the following databases: EMBASE, MEDLINE, HMIC, PsycInfo, and Cochrane (April 1964 to February 12, 2019) without language restriction. The search was limited to articles reporting the use of electronic alerts in adult medical inpatients with sepsis who were not initially in intensive care. The search terms included digital technology and sepsis (for complete search strategy, see the Multimedia Appendix 1 search strategy). There were supplemental references that were manually gathered through reference lists.

Our primary outcome was hospital and intensive care unit (ICU) length of stay (LOS) for patients with sepsis. Secondary outcome measures included sepsis-associated mortality and the time to antibiotics.

Studies were eligible for inclusion if patients had sepsis with a clear reporting of the diagnostic criteria. In addition, studies were only eligible for inclusion if the outcomes could be compared pre- and postintroduction of digital alerts for sepsis. Studies where the clear outcome measures of interest were not stated were not included. The outcomes of interest were mortality, time to antibiotics, and ICU LOS, and hospital LOS. Studies only required one outcome of interest to be eligible for inclusion. There was no fixed number of patients required in each treatment group nor a fixed duration of follow-up.

All included study characteristics and outcome measures were extracted by one investigator and verified by another. All full-text reports of studies identified as potentially eligible after title and abstract review were obtained for further review. All empirical research into digital sepsis alerting in adults and outcome measures were included. Information on the study location, type of hospital, type of alert, and whom the alert was directed at were all extracted using a set standardized form. Studies were only included if a comparative study of pre- and postalerting groups was performed and the number of patients in each group was clearly stated. Studies solely based on intensive care were excluded, as extra levels of care and treatments offered to these patients would affect the sepsis outcome measures. Additionally, case reports, narrative reviews, editorials, and commentaries were excluded. After full text review, if a study was excluded, the reasons were noted and disagreements were resolved by a third reviewer. A PRISMA flow diagram was used to map studies included, studies excluded, and the reasons for their inclusion/exclusion (Multimedia Appendix 2).

A qualitative assessment of all studies was performed. For the primary outcome of hospital LOS, a weighted mean difference was calculated. A weighted mean difference was also used in ICU LOS. For the secondary outcome measure of mortality, we used percentage change between the pre- and postalerting groups. To ensure a standard way to compare the studies, all times to antibiotics were converted into minutes and number of days used for ICU/hospital LOS. For the meta-analysis, outcomes were analyzed by calculating the difference in means between studies or the ratio of means within each study. We substituted the median for the mean in studies where only the median was reported. We employed an inverse-variance approach with a random effects model using the DerSimonian and Laird methodology for both continuous and categorical variables. This was accomplished using Stata 13 (StataCorp, College Station, Texas). The I² statistic was used to estimate the degree of heterogeneity between studies.

The Newcastle-Ottawa scale (NOS) was used to score the included papers on the basis of the quality [19]. The range of scores was from 0 to 9. Studies with a score of ≥7 were rated as higher-quality studies, whereas those with scores ≤ 7 were rated as lower-quality studies.

The literature search identified 3861 references (appendix). One article was found through hand searching. After screening, 72 full-text reviews were reviewed, including one translated from Spanish. A total of 16 studies met the inclusion criteria and were included in the final analysis. A description of the studies including the sepsis intervention, alert type, and criteria used to define sepsis can be found in Multimedia Appendix 3. All studies that defined sepsis alerts and the pre- and postalerting stages are presented in Multimedia Appendix 4.

All studies investigated digital sepsis alerts (Multimedia Appendix 3). Several studies additionally used a rapid response team in combination with the digital sepsis alert in the alerting group [20-22]. An electronic order set was used by four studies in the alerting group [21,23-25].

The studies were predominantly single-center studies; however, one study that was multicenter study across three hospital sites [26]. Most studies were based in the United States; however, one study was from the Kingdom of Saudi Arabia [20] and one was from Spain [27]. Most studies were based in large hospital settings; six studies were based in academic centers [20,22,23,28-30], and three centers were level 1 trauma centers [21,29,31]. Within the hospital setting, only one study had a hospital-wide sepsis alert [21], and eight had alerts solely in the emergency department (ED) [20,23,27,29-33]. In addition, five studies included only medical wards [24,26,28,34,35], and two studies included both on medical and surgical wards [22,25]. All the studies were pre- and postimplementation studies.

The type of electronic alerts used in the studies varied. Most studies used digital alerts generated through EHRs on desktop-based computers [20-23,27,29-35]. Two studies involved alerting via a mobile device [24,25] and two studies used text paging [26,28]. One study predominantly used alerting through EHRs but used additional text paging at the discretion of the ED attending [23]. This is shown in Multimedia Appendix 3. Most of the digital alerts in the study were aimed at the nursing staff [20,22-25,28,32]. Although one study was aimed solely at clinicians [31], two studies had alerts sent to both clinicians and the rapid response team [21,26]. In addition, in four studies, it was unknown whom the digital alerts were sent to [27,29,33,35].

The sepsis-alerting criteria varied among studies; many studies used a variation of the SIRS criteria [20-24,26,29,34]. SIRS is defined as a temperature >38°C or <36°C, heart rate >90 beats/min, respiratory rate >20 breaths/min or arterial CO₂ lower than 32 mm Hg, and white blood cell count >12,000/mm³ or <4000/mm³ [16]. Only one study used two or more SIRS criteria alone [34]. Some used two or more SIRS criteria with a suspicion of infection [21,23,31]. Hayden et al used two or more SIRS criteria with a suspicion of infection but had two subgroups of patients: one with a systolic blood pressure (SBP) of <90 mm Hg and one with an SBP >90 mm Hg. A combination of two or more SIRS criteria with evidence of end-organ dysfunction was also used as a criterion to alert [20,26,30]. Narayanan et al reported two types of sepsis alerts: one alert if two or more SIRS criteria were used and another if severe sepsis/septic shock criteria (ie, two or more SIRS criteria + end-organ dysfunction OR fluid nonresponsive hypotension) [30]. One study used the combination of two or more SIRS criteria and anion gap acidosis [22]. Crum et al used a combination of two or more SIRS criteria; infectious ED diagnosis or symptom diagnosis and antibiotic administration in the ED; and acute organ dysfunction (lactate level ≥4.0 mmol/L and persistent hypotension within 6 hours of triage or vasopressor use) [29]. Austrian et al had two types of alerts: one with two or more SIRS criteria and another with an SBP <90 mm Hg or lactate level ≥ 4 mg/dL. Not all studies used SIRS; Pulia et al [33] used a fever plus any abnormal vital sign, whereas Sawyer et al [28] and Ferreras et al [27] used an algorithm consisting of laboratory values and hemodynamic parameters. The study by Mathews et al [35] was an abstract only and no details were found on the alerting criteria used.

All studies included had clear outcome measures with a defined sepsis alert and pre- and postalerting stage (Multimedia Appendix 4). Some studies reviewed several outcome measures of interest. All studies were non-ICU based and either alerted in the ED or the hospital wards. In total, 8 studies reviewed the hospital LOS, 5 studies reviewed the ICU LOS, 11 reviewed the mortality outcomes, and 5 studies reviewed the time to antibiotics. Several studies have tried to adjust for potential confounders. Guirgis et al [21] performed multivariate adjusted comparisons, whereas Westra [25] used techniques such as propensity scoring and bootstrapping to adjust for confounders [25]. Each outcome measure is reviewed below in further detail.

Hospital LOS was included in eight studies [20,21,23,25,26,28,32,34]. In total, there were 3948 patients in the prealerting group and 4872 patients in the postalerting group across these eight studies. The results highlight a significant reduction in hospital LOS by 1.311 days in the postalerting group (95% CI –2.362 to –0.261; P=.014). The proportional decrease in hospital LOS was the study with the biggest impact on LOS, which was Arabi et al [20]. The strong treatment effect seen may be partly explained by the addition of a rapid response team; however, the rapid response team was also used in the study by Guirgis et al [21] and the effect on LOS was not as remarkable.

For ICU LOS (when patients were escalated from the ward setting), five studies were included [20,21,23,26,32] (Multimedia Appendix 4). In total, 3627 patients were included in the prealerting group and 4475 were included in the postalerting group. The weighted mean difference for all studies showed a significant reduction in ICU LOS by 0.766 days for the alerting group versus the prealerting group (95% CI –1.324 to 0.209; P=.007). The biggest impacting study was the one by Arabi et al [20]. This may have been explained by the addition of a rapid response team; however, the rapid response team was also used in the study by Guirgis et al [21] and the effect on LOS was not as remarkable.

For mortality, 11 studies were included [20,24,27,28,31,32,34,35]. In total, there were 5868 patients in the prealerting group and 6629 in the postalerting group. There was no significant reduction between digital alerts and mortality (mean decrease 11.4%; 95% CI –0.873 to 0.646; P=.77).

For time to antibiotics, five studies were included [20,23,29,33] (Multimedia Appendix 4). In total, there were 991 patients in the prealerting group and 1473 in the postalerting group. On the basis of three studies with available data, the weighted mean difference in time to antibiotics from the prealerting to the alerting group showed no significant reduction (126 min) in time to antibiotics (95% CI -291.113 to 39.015; P=.13).

The diagnostic accuracy of the digital sepsis alerts is shown in Multimedia Appendix 5. Of all the abovementioned studies included in the review, only five studies attempted to address the diagnostic accuracy [24,32]. There was high heterogeneity in the accuracy of alerting in these studies. The specificity of digital alerting in sepsis remained high, with four studies reporting specificities of 81.92% (78.73-84.8 range) [24], 97% [26], 82.0% [25], and 94.56% (93.64-95.32 range) [27]. Although the specificity remained high, the sensitivity showed high heterogeneity: 95.16% (89.77-98.20) [24], 95.2% [25], 87% (81.93-91.66) [27], 80.4% [32], and 16% [26]. The sensitivities and specificities were not given in any other studies. In total, three studies had high negative predictive values of digital alerting and sepsis ranging from 94% [26] to 98.88% (97.58-99.59) [24] and 99.11% (98.69-99.4) [27]. The positive predictive value was much lower at 14.6% [32], 26% [26], 50.21% (43.64-56.78) [24], and 51.64% (46.15-57.11) [27].

All included studies were assessed for the quality of their methodology. All studies had a nonrandomized design methodology, and the NOS was used (Multimedia Appendix 6). Of all studies included, only six were of higher quality. Many had poor comparability due to a lack of randomization.