Published on in Vol 21, No 12 (2019): December

Preprints (earlier versions) of this paper are available at, first published .
Digital Alerting and Outcomes in Patients With Sepsis: Systematic Review and Meta-Analysis

Digital Alerting and Outcomes in Patients With Sepsis: Systematic Review and Meta-Analysis

Digital Alerting and Outcomes in Patients With Sepsis: Systematic Review and Meta-Analysis


1Chelsea and Westminster Hospital, NHS Foundation Trust, London, United Kingdom

2Department of Surgery and Cancer, Imperial College London, London, United Kingdom

3Division of Infectious Diseases, Imperial College London, London, United Kingdom

Corresponding Author:

Meera Joshi, MBBS, MRCS

Chelsea and Westminster Hospital

NHS Foundation Trust

369 Fulham Rd, Chelsea

London, SW10 9NH

United Kingdom

Phone: 44 2033158000


Background: The diagnosis and management of sepsis remain a global health care challenge. Digital technologies have the potential to improve sepsis care.

Objective: The aim of this paper was to systematically review the evidence on the impact of digital alerting systems on sepsis related outcomes.

Methods: The following databases were searched for studies published from April 1964 to February 12, 2019, with no language restriction: EMBASE, MEDLINE, HMIC, PsycINFO, and Cochrane. All full-text reports of studies identified as potentially eligible after title and abstract reviews were obtained for further review. The search was limited to adult inpatients. Relevant articles were hand searched for other studies. Only studies with clear pre- and postalerting phases were included. Primary outcomes were hospital length of stay (LOS) and intensive care LOS, whereas secondary outcomes were time to antibiotics and mortality. Studies based solely on intensive care, case reports, narrative reviews, editorials, and commentaries were excluded. All other trial designs were included. A qualitative assessment and meta-analysis were performed.

Results: This review identified 72 full-text articles. From these, 16 studies met the inclusion criteria and were included in the final analysis. Of these, 8 studies reviewed hospital LOS, 12 reviewed mortality outcomes, 5 studies explored time to antibiotics, and 5 studies investigated intensive care unit (ICU) LOS. Both quantitative and qualitative assessments of the studies were performed. There was evidence of a significant benefit of digital alerting in hospital LOS, which reduced by 1.31 days (P=.014), and ICU LOS, which reduced by 0.766 days (P=.007). There was no significant association between digital alerts and mortality (mean decrease 11.4%; P=.77) or time to antibiotics (mean decrease 126 min; P=.13).

Conclusions: This review highlights that digital alerts can considerably reduce hospital and ICU stay for patients with sepsis. Further studies including randomized controlled trials are necessary to confirm these findings and identify the choice of alerting system according to the patient status and pathological cohort.

J Med Internet Res 2019;21(12):e15166



Sepsis Challenges

Sepsis is a major health concern causing significant mortality worldwide [1,2]. Although used to describe a variety of clinical conditions, the most recent recommendations suggest that sepsis should be defined as a “life threatening organ dysfunction caused by a dysregulated host response to infection” [3]. In the United States, the average annual age adjusted incidence is 300-1000 cases per 100,000 people [4]. The mortality from this condition is as high as one in four patients [1,2,5,6]. In addition, it poses a significant financial burden on health care systems. The annual total costs of sepsis in the United States was US $20 billion in 2011 [7]. Additionally, there are numerous indirect costs that may significantly impact patients’ quality of life. For example, the older population may have severe long-term health problems after sepsis, including both cognitive dysfunction and functional disability [8].

Early Detection of Sepsis

Recognizing sepsis early and initiating timely treatment results in improved patient outcomes and significant cost reduction [9,10]. The International Surviving Sepsis Campaign recommends early identification of patients with sepsis and immediate treatment such as antibiotics within 1 hour of suspecting sepsis and septic shock [11]. A retrospective multicenter study found for every hour’s delay in the treatment of patients with septic shock, the risk of death increases by 7.6% [12]. Delays in identification of sepsis are often attributable to a lack of diagnostic tools, gradual disease progression, and no gold standard for diagnosis [13]. Interventions such as regular monitoring of vital signs and elevated lactate levels aid early recognition [14]. However, despite these measures, patients are still being diagnosed late. Although track and trigger scoring systems such as the National Early Warning Score have standardized the documentation of vital signs and enabled earlier recognition to a degree, they are still subject to inaccuracies in recordings because of their subjective nature and intermittent monitoring [15]. Alternate strategies in sepsis detection are therefore urgently required to improve outcomes.

Digital Technology in Sepsis

Digital technology holds significant promise in enabling early sepsis recognition. Coupled with the rapid expansion of electronic health records (EHRs) worldwide, it is now possible to provide health care staff with real-time information on laboratory tests, imaging, and physiological vital signs at the bedside. This is particularly relevant in sepsis because of the traditional reliance on risk scoring systems used to define sepsis. These include the Systemic Inflammatory Response Syndrome (SIRS) and Sequential Organ Failure Assessment (SOFA) score [16,17]. Although usually calculated through paper-based observational charts alongside separate interfaces to retrieve blood test results, both have the potential to be automated using EHRs to reduce error and enhance digital alerting. There is now increased literature in this field that can support new clinical decisions in sepsis management. The aim of our paper was to offer an up-to-date systematic review of digital alerting systems on patient sepsis outcomes.

Data Sources and Searches

We applied the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for this review [18]. Our search on February 12, 2019, included the following databases: EMBASE, MEDLINE, HMIC, PsycInfo, and Cochrane (April 1964 to February 12, 2019) without language restriction. The search was limited to articles reporting the use of electronic alerts in adult medical inpatients with sepsis who were not initially in intensive care. The search terms included digital technology and sepsis (for complete search strategy, see the Multimedia Appendix 1 search strategy). There were supplemental references that were manually gathered through reference lists.

Outcome Measures

Our primary outcome was hospital and intensive care unit (ICU) length of stay (LOS) for patients with sepsis. Secondary outcome measures included sepsis-associated mortality and the time to antibiotics.

Study Selection

Studies were eligible for inclusion if patients had sepsis with a clear reporting of the diagnostic criteria. In addition, studies were only eligible for inclusion if the outcomes could be compared pre- and postintroduction of digital alerts for sepsis. Studies where the clear outcome measures of interest were not stated were not included. The outcomes of interest were mortality, time to antibiotics, and ICU LOS, and hospital LOS. Studies only required one outcome of interest to be eligible for inclusion. There was no fixed number of patients required in each treatment group nor a fixed duration of follow-up.

Data Extraction and Quality Assessment

All included study characteristics and outcome measures were extracted by one investigator and verified by another. All full-text reports of studies identified as potentially eligible after title and abstract review were obtained for further review. All empirical research into digital sepsis alerting in adults and outcome measures were included. Information on the study location, type of hospital, type of alert, and whom the alert was directed at were all extracted using a set standardized form. Studies were only included if a comparative study of pre- and postalerting groups was performed and the number of patients in each group was clearly stated. Studies solely based on intensive care were excluded, as extra levels of care and treatments offered to these patients would affect the sepsis outcome measures. Additionally, case reports, narrative reviews, editorials, and commentaries were excluded. After full text review, if a study was excluded, the reasons were noted and disagreements were resolved by a third reviewer. A PRISMA flow diagram was used to map studies included, studies excluded, and the reasons for their inclusion/exclusion (Multimedia Appendix 2).

Data Synthesis and Analysis

A qualitative assessment of all studies was performed. For the primary outcome of hospital LOS, a weighted mean difference was calculated. A weighted mean difference was also used in ICU LOS. For the secondary outcome measure of mortality, we used percentage change between the pre- and postalerting groups. To ensure a standard way to compare the studies, all times to antibiotics were converted into minutes and number of days used for ICU/hospital LOS. For the meta-analysis, outcomes were analyzed by calculating the difference in means between studies or the ratio of means within each study. We substituted the median for the mean in studies where only the median was reported. We employed an inverse-variance approach with a random effects model using the DerSimonian and Laird methodology for both continuous and categorical variables. This was accomplished using Stata 13 (StataCorp, College Station, Texas). The I2 statistic was used to estimate the degree of heterogeneity between studies.

Risk-of-Bias Assessment

The Newcastle-Ottawa scale (NOS) was used to score the included papers on the basis of the quality [19]. The range of scores was from 0 to 9. Studies with a score of ≥7 were rated as higher-quality studies, whereas those with scores ≤ 7 were rated as lower-quality studies.

Search Results

The literature search identified 3861 references (appendix). One article was found through hand searching. After screening, 72 full-text reviews were reviewed, including one translated from Spanish. A total of 16 studies met the inclusion criteria and were included in the final analysis. A description of the studies including the sepsis intervention, alert type, and criteria used to define sepsis can be found in Multimedia Appendix 3. All studies that defined sepsis alerts and the pre- and postalerting stages are presented in Multimedia Appendix 4.

Characteristics of Studies and Digital Alerting Used

All studies investigated digital sepsis alerts (Multimedia Appendix 3). Several studies additionally used a rapid response team in combination with the digital sepsis alert in the alerting group [20-22]. An electronic order set was used by four studies in the alerting group [21,23-25].

The studies were predominantly single-center studies; however, one study that was multicenter study across three hospital sites [26]. Most studies were based in the United States; however, one study was from the Kingdom of Saudi Arabia [20] and one was from Spain [27]. Most studies were based in large hospital settings; six studies were based in academic centers [20,22,23,28-30], and three centers were level 1 trauma centers [21,29,31]. Within the hospital setting, only one study had a hospital-wide sepsis alert [21], and eight had alerts solely in the emergency department (ED) [20,23,27,29-33]. In addition, five studies included only medical wards [24,26,28,34,35], and two studies included both on medical and surgical wards [22,25]. All the studies were pre- and postimplementation studies.

The type of electronic alerts used in the studies varied. Most studies used digital alerts generated through EHRs on desktop-based computers [20-23,27,29-35]. Two studies involved alerting via a mobile device [24,25] and two studies used text paging [26,28]. One study predominantly used alerting through EHRs but used additional text paging at the discretion of the ED attending [23]. This is shown in Multimedia Appendix 3. Most of the digital alerts in the study were aimed at the nursing staff [20,22-25,28,32]. Although one study was aimed solely at clinicians [31], two studies had alerts sent to both clinicians and the rapid response team [21,26]. In addition, in four studies, it was unknown whom the digital alerts were sent to [27,29,33,35].

The sepsis-alerting criteria varied among studies; many studies used a variation of the SIRS criteria [20-24,26,29,34]. SIRS is defined as a temperature >38°C or <36°C, heart rate >90 beats/min, respiratory rate >20 breaths/min or arterial CO2 lower than 32 mm Hg, and white blood cell count >12,000/mm3 or <4000/mm3 [16]. Only one study used two or more SIRS criteria alone [34]. Some used two or more SIRS criteria with a suspicion of infection [21,23,31]. Hayden et al used two or more SIRS criteria with a suspicion of infection but had two subgroups of patients: one with a systolic blood pressure (SBP) of <90 mm Hg and one with an SBP >90 mm Hg. A combination of two or more SIRS criteria with evidence of end-organ dysfunction was also used as a criterion to alert [20,26,30]. Narayanan et al reported two types of sepsis alerts: one alert if two or more SIRS criteria were used and another if severe sepsis/septic shock criteria (ie, two or more SIRS criteria + end-organ dysfunction OR fluid nonresponsive hypotension) [30]. One study used the combination of two or more SIRS criteria and anion gap acidosis [22]. Crum et al used a combination of two or more SIRS criteria; infectious ED diagnosis or symptom diagnosis and antibiotic administration in the ED; and acute organ dysfunction (lactate level ≥4.0 mmol/L and persistent hypotension within 6 hours of triage or vasopressor use) [29]. Austrian et al had two types of alerts: one with two or more SIRS criteria and another with an SBP <90 mm Hg or lactate level ≥ 4 mg/dL. Not all studies used SIRS; Pulia et al [33] used a fever plus any abnormal vital sign, whereas Sawyer et al [28] and Ferreras et al [27] used an algorithm consisting of laboratory values and hemodynamic parameters. The study by Mathews et al [35] was an abstract only and no details were found on the alerting criteria used.

All studies included had clear outcome measures with a defined sepsis alert and pre- and postalerting stage (Multimedia Appendix 4). Some studies reviewed several outcome measures of interest. All studies were non-ICU based and either alerted in the ED or the hospital wards. In total, 8 studies reviewed the hospital LOS, 5 studies reviewed the ICU LOS, 11 reviewed the mortality outcomes, and 5 studies reviewed the time to antibiotics. Several studies have tried to adjust for potential confounders. Guirgis et al [21] performed multivariate adjusted comparisons, whereas Westra [25] used techniques such as propensity scoring and bootstrapping to adjust for confounders [25]. Each outcome measure is reviewed below in further detail.

Hospital Length of Stay

Hospital LOS was included in eight studies [20,21,23,25,26,28,32,34]. In total, there were 3948 patients in the prealerting group and 4872 patients in the postalerting group across these eight studies. The results highlight a significant reduction in hospital LOS by 1.311 days in the postalerting group (95% CI –2.362 to –0.261; P=.014). The proportional decrease in hospital LOS was the study with the biggest impact on LOS, which was Arabi et al [20]. The strong treatment effect seen may be partly explained by the addition of a rapid response team; however, the rapid response team was also used in the study by Guirgis et al [21] and the effect on LOS was not as remarkable.

Intensive Care Unit Length of Stay

For ICU LOS (when patients were escalated from the ward setting), five studies were included [20,21,23,26,32] (Multimedia Appendix 4). In total, 3627 patients were included in the prealerting group and 4475 were included in the postalerting group. The weighted mean difference for all studies showed a significant reduction in ICU LOS by 0.766 days for the alerting group versus the prealerting group (95% CI –1.324 to 0.209; P=.007). The biggest impacting study was the one by Arabi et al [20]. This may have been explained by the addition of a rapid response team; however, the rapid response team was also used in the study by Guirgis et al [21] and the effect on LOS was not as remarkable.


For mortality, 11 studies were included [20,24,27,28,31,32,34,35]. In total, there were 5868 patients in the prealerting group and 6629 in the postalerting group. There was no significant reduction between digital alerts and mortality (mean decrease 11.4%; 95% CI –0.873 to 0.646; P=.77).

Time to Antibiotics

For time to antibiotics, five studies were included [20,23,29,33] (Multimedia Appendix 4). In total, there were 991 patients in the prealerting group and 1473 in the postalerting group. On the basis of three studies with available data, the weighted mean difference in time to antibiotics from the prealerting to the alerting group showed no significant reduction (126 min) in time to antibiotics (95% CI -291.113 to 39.015; P=.13).

Diagnostic Accuracy

The diagnostic accuracy of the digital sepsis alerts is shown in Multimedia Appendix 5. Of all the abovementioned studies included in the review, only five studies attempted to address the diagnostic accuracy [24,32]. There was high heterogeneity in the accuracy of alerting in these studies. The specificity of digital alerting in sepsis remained high, with four studies reporting specificities of 81.92% (78.73-84.8 range) [24], 97% [26], 82.0% [25], and 94.56% (93.64-95.32 range) [27]. Although the specificity remained high, the sensitivity showed high heterogeneity: 95.16% (89.77-98.20) [24], 95.2% [25], 87% (81.93-91.66) [27], 80.4% [32], and 16% [26]. The sensitivities and specificities were not given in any other studies. In total, three studies had high negative predictive values of digital alerting and sepsis ranging from 94% [26] to 98.88% (97.58-99.59) [24] and 99.11% (98.69-99.4) [27]. The positive predictive value was much lower at 14.6% [32], 26% [26], 50.21% (43.64-56.78) [24], and 51.64% (46.15-57.11) [27].

Risk of Bias Assessment

All included studies were assessed for the quality of their methodology. All studies had a nonrandomized design methodology, and the NOS was used (Multimedia Appendix 6). Of all studies included, only six were of higher quality. Many had poor comparability due to a lack of randomization.

Patients with sepsis have high morbidity, mortality, and associated treatment costs [1,7,36].

This review found that digital alerting in sepsis is associated with significant reductions in hospital LOS by 1.311 days (P=.014) and ICU LOS by 0.766 days (P=.007). Both mortality and time to antibiotics showed no significant differences between the prealerting and postalerting groups.

Reduced hospital LOS and ICU LOS are likely due to earlier diagnosis, improved time to treatments such as antibiotics, earlier fluid replacement, minimized inflammatory cytokines, and cytokine stress. A previous systematic review on the diagnostic accuracy and effectiveness of digital alerts found no improvements in clinical outcomes when digital alerts in sepsis were employed [37]. However, since its publication, there has been a growth in the number of empirical studies on digital alerting and sepsis outcomes, as more hospitals worldwide are transferring to EHRs and digital solutions. All studies identified in this review are recent and after 2010, reflecting the entry of new technology in this field.

This review also identified a wide variation in the diagnosis of sepsis and multiple measures used for this purpose. Many of the studies included in the review have used the SIRS definition of sepsis or a modification of it. This is perhaps because it is the most practical in a clinical setting with readily available vital sign data and laboratory tests. Although SIRS was routinely used as a definition of sepsis, the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis 3) declared that the use of two or more SIRS criteria in identifying sepsis was not specific enough [3]. The SIRS criteria may be met without infective disease processes, such as trauma [38]. The variation in diagnostic criteria may explain the lack of literature meta-analysis in digital sepsis alerts to date. The ideal alerting criteria required for sepsis in combination with guidance on the latest definitions must be reviewed in further large-scale randomized studies. The ideal sepsis alerts must correctly identify patients with sepsis and exclude those without sepsis. Alerting with low positive predictive value is likely to contribute to alert fatigue, and it is suggested that more sophisticated algorithms may be required to correctly identify patients with sepsis [32]. A recent retrospective comparison of scoring systems found the National Early Warning Score to be more accurate in detecting sepsis as compared with other scoring systems including SIRS [39]. Furthermore, multicenter studies are required to review these findings. In addition to the alert itself, the way the alert is delivered must be reviewed. There is no one best method in the literature to the type of alert to be used and who it needs to be sent to. A combination of EHR, text paging, and mobile alerts have been used in the studies included in this review.

Of all the studies included in the review, only five studies attempted to address diagnostic accuracy [24,32]. There was high heterogeneity in the accuracy of the alerting in these studies. Far more research is required in large studies to elucidate diagnostic accuracy in digital alerting. Although digital alerting in sepsis is beneficial, it is likely that the alerts be used in combination with an application of other resources such as staff education, direct communication for patients with sepsis, and rapid response teams. In one study, 5 months of training in addition to the sepsis alert helped contribute to a reduction in sepsis mortality [24].

Despite the significance of some of the clinical outcomes assessed, our study has limitations due to the variability of methodologies and study types, and most studies scored low on quality. Most of the studies are observational in design, thus creating a high risk of bias. There was a high heterogeneity between the studies, and there were very limited data on the diagnostic accuracy of many of the included results. Furthermore, the pre- and postalerting study design may be prone to selection bias due to variations in the patient population and changes in ways of providing care at hospitals over time. A further limitation of the studies is that the size of the postalerting cohort is larger than that of the prealerting cohort, despite the fact that many of the included studies followed postimplementation cohorts for a shorter period of time than the preimplementation cohort. Although all studies used electronic digital alerting, some studies additionally used a rapid response team and electronic order sets. This may have led to a confounding bias in outcomes.

There are several studies published recently with different study designs to help address some of these concerns. The first is a retrospective cohort study with patients exposed or not exposed to the electronic ICU telemedicine sepsis management in the ED [40]. This is a novel form of alerting using telemedicine. In this study, the exposed cohort had a quicker time to antibiotics than the unexposed cohort (122.3 [SD 83.3] min versus 163.4 [SD 204.4] min; P=.04] [40]. However, the hospital LOS and mortality were similar between both cohorts.

Another study design is the use of a prospective quality improvement study [41]. The alerting system was customized to local practice with regularly incorporated feedback through the plan-do-study-act (PDSA) cycles [41]. In this study, a machine learning algorithm was implemented with continuous incorporation of feedback, creating a system tailor made to the hospital workflow [41]. After several PDSA cycles, the machine learning algorithm sepsis mortality reduced by 60.24% with a hospital LOS reduction of 9.55% [41]. The alerting in this study was very different from that in other studies because of the machine learning approach [41]. Although it was not included in the original meta-analysis, the analysis was redone to include this study, and there was no difference in either the hospital LOS (proportional decrease: 11.3%; 95% CI –0.189 to 0.038; P=.003) or mortality (mean decrease: 11.4%; 95% CI –0.844 to 0.616; P=.76).

Further research should also seek to answer several other important questions such as the optimal type of sepsis alert, which team members should be alerted, and at what frequency should they be alerted. A recent observational cohort study reviewing successive improvements over a 10-year period found that reviewed sepsis alerts were sent to a telephonist and alerts were sent to a nurse’s mobile phone [42]. They found that time to antibiotics was reduced to 1 hour (55 min to 1 hour 30 min) when the alert was sent to the telephonist and to 45 min (30 min to 1 hour) when the alert was sent directly to the nurse’s mobile phone (P=.02) [42].

Further large-scale multicenter prospective randomized controlled trials (RCTs) with a single standardized sepsis alert are required. An RCT would help exclude confounding biases, although implementation maybe challenging. Overall, two recent RCTs have been published; however, both studies had insufficient sample sizes, with insufficient power to detect differences between the prealerting and postalerting groups [43,44]. The results of both RCTs were also contradictory. One RCT on sepsis alerting through EHRs found no significant difference in outcome measures after alerting [43]. However, at least 66% of patients were on antibiotics at the time of the alert, and this high baseline compliance may have led to a lower marginal return on improved detection through alerting and partly explain these findings [43]. The other RCT on the use of machine learning–based sepsis prediction systems supported similar findings to this review with a reduction in hospital LOS by 2.7 days and a mortality reduction of 58% [44].

The cumulative effect of an additional rapid response team, staff education, and electronic order sets alongside digital alerting must also be evaluated. Further studies should also be performed in different clinical health systems, as the value of alerting is likely to depend on baseline performance of health systems. This would aid generalizability and widespread implementation. Importantly, the diagnostic accuracy for the digital alerting in most studies included is unknown and needs to be clearly defined in future work to ascertain the validity of findings. Finally, the cost-effectiveness of digital sepsis alerts also needs further evaluation if we are to justify its use in the current fiscal climate.

Automated digital alerts can improve sepsis-related outcomes. This review highlights a significant reduction in hospital LOS by 1.311 days and ICU LOS by 0.766 days. The emergence of digital technologies has the capacity to transform the rapid identification of patient physiological deterioration and, in turn, revolutionize patient care through alerts and novel treatment innovations (such as smart wards). Higher-quality evidence through larger, better-designed randomized studies is needed to guide the application of digital alerting in patients at risk of sepsis.


We thank Mrs Jackie Kemp, the librarian, for her assistance with the search and reviewing of the databases. This paper is an independent research funded by the National Institute for Health Research (NIHR) Imperial Biomedical Research Centre (BRC) and NIHR Imperial Patient Safety Translational Research Centre (PSTRC). The views expressed in this publication are those of the author(s) and not necessarily those of the National Health Service, the NIHR, or the Department of Health and Social Care. Infrastructure support for this research was provided by the NIHR Imperial BRC and the NIHR Imperial PSTRC.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Search strategy.

DOCX File , 14 KB

Multimedia Appendix 2

Preferred Reporting Items for Systematic Reviews and Meta-analyses flow diagram.

DOCX File , 38 KB

Multimedia Appendix 3

Characterization of studies and digital alerts used.

DOCX File , 23 KB

Multimedia Appendix 4

All studies reviewing outcome measures.

DOCX File , 21 KB

Multimedia Appendix 5

Diagnostic accuracy.

DOCX File , 20 KB

Multimedia Appendix 6

Risk of bias of a nonrandomized prospective comparative cohort comparing pre and postalerting based on the Newcastle-Ottawa Scale.

DOCX File , 15 KB

  1. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001 Jul;29(7):1303-1310. [CrossRef] [Medline]
  2. Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med 2003 Apr 17;348(16):1546-1554. [CrossRef] [Medline]
  3. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). J Am Med Assoc 2016 Mar 23;315(8):762-774 [FREE Full text] [CrossRef] [Medline]
  4. Kempker JA, Martin GS. The changing epidemiology and definitions of sepsis. Clin Chest Med 2016 Jun;37(2):165-179 [FREE Full text] [CrossRef] [Medline]
  5. Linde-Zwirble WT, Angus DC. Severe sepsis epidemiology: sampling, selection, and society. Crit Care 2004 Aug;8(4):222-226 [FREE Full text] [CrossRef] [Medline]
  6. Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med 2007 May;35(5):1244-1250. [CrossRef] [Medline]
  7. Torio C, Andrews R. Agency for Healthcare Research and Quality. 2013. National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2011   URL: [accessed 2019-10-29]
  8. Iwashyna TJ, Ely EW, Smith DM, Langa KM. Long-term cognitive impairment and functional disability among survivors of severe sepsis. J Am Med Assoc 2010 Oct 27;304(16):1787-1794 [FREE Full text] [CrossRef] [Medline]
  9. Gaieski DF, Mikkelsen ME, Band RA, Pines JM, Massone R, Furia FF, et al. Impact of time to antibiotics on survival in patients with severe sepsis or septic shock in whom early goal-directed therapy was initiated in the emergency department. Crit Care Med 2010 Apr;38(4):1045-1053. [CrossRef] [Medline]
  10. Paoli CJ, Reynolds MA, Sinha M, Gitlin M, Crouser E. Epidemiology and costs of sepsis in the United States-an analysis based on timing of diagnosis and severity level. Crit Care Med 2018 Dec;46(12):1889-1897 [FREE Full text] [CrossRef] [Medline]
  11. Rhodes A, Evans LE, Alhazzani W, Levy MM, Antonelli M, Ferrer R, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Intensive Care Med 2017 Mar;43(3):304-377. [CrossRef] [Medline]
  12. Kumar A, Roberts D, Wood KE, Light B, Parrillo JE, Sharma S, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit Care Med 2006 Jun;34(6):1589-1596. [CrossRef] [Medline]
  13. McLymont N, Glover GW. Scoring systems for the characterization of sepsis and associated outcomes. Ann Transl Med 2016 Dec;4(24):527 [FREE Full text] [CrossRef] [Medline]
  14. Lee SM, An WS. New clinical criteria for septic shock: serum lactate level as new emerging vital sign. J Thorac Dis 2016 Jul;8(7):1388-1390 [FREE Full text] [CrossRef] [Medline]
  15. Downey C, Tahir W, Randell R, Brown J, Jayne D. Strengths and limitations of early warning scores: a systematic review and narrative synthesis. Int J Nurs Stud 2017 Nov;76:106-119. [CrossRef] [Medline]
  16. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest 1992 Jun;101(6):1644-1655. [CrossRef] [Medline]
  17. Vincent J, de Mendonça A, Cantraine F, Moreno R, Takala J, Suter P, et al. Use of the SOFA score to assess the incidence of organ dysfunction/failure in intensive care units: results of a multicenter, prospective study. Working group on "sepsis-related problems" of the European Society of Intensive Care Medicine. Crit Care Med 1998 Nov;26(11):1793-1800. [CrossRef] [Medline]
  18. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J 2009 Jul 21;339:b2535 [FREE Full text] [CrossRef] [Medline]
  19. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol 2010 Sep;25(9):603-605. [CrossRef] [Medline]
  20. Arabi YM, Al-Dorzi HM, Alamry A, Hijazi R, Alsolamy S, Al Salamah M, et al. The impact of a multifaceted intervention including sepsis electronic alert system and sepsis response team on the outcomes of patients with sepsis and septic shock. Ann Intensive Care 2017 Dec;7(1):57 [FREE Full text] [CrossRef] [Medline]
  21. Guirgis FW, Jones L, Esma R, Weiss A, McCurdy K, Ferreira J, et al. Managing sepsis: electronic recognition, rapid response teams, and standardized care save lives. J Crit Care 2017 Aug;40:296-302 [FREE Full text] [CrossRef] [Medline]
  22. Benson L, Hasenau S, O'Connor N, Burgermeister D. The impact of a nurse practitioner rapid response team on systemic inflammatory response syndrome outcomes. Dimens Crit Care Nurs 2014;33(3):108-115. [CrossRef] [Medline]
  23. Hayden GE, Tuuri RE, Scott R, Losek JD, Blackshaw AM, Schoenling AJ, et al. Triage sepsis alert and sepsis protocol lower times to fluids and antibiotics in the ED. Am J Emerg Med 2016 Jan;34(1):1-9 [FREE Full text] [CrossRef] [Medline]
  24. Manaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inform Assoc 2017 Jan;24(1):88-95. [CrossRef] [Medline]
  25. Westra B, Landman S, Yadav P, Steinbach M. Secondary analysis of an electronic surveillance system combined with multi-focal interventions for early detection of sepsis. Appl Clin Inform 2017 Jan 18;8(1):47-66 [FREE Full text] [CrossRef] [Medline]
  26. Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, et al. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015 Jan;10(1):26-31 [FREE Full text] [CrossRef] [Medline]
  27. Ferreras J, Judez D, Tirado G, Aspiroz C, Martínez-Álvarez R, Dorado P, et al. [Implementation of an automatic alarms system for early detection of patients with severe sepsis]. Enferm Infecc Microbiol Clin 2015 Oct;33(8):508-515. [CrossRef] [Medline]
  28. Sawyer AM, Deal EN, Labelle AJ, Witt C, Thiel SW, Heard K, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med 2011 Mar;39(3):469-473. [CrossRef] [Medline]
  29. Crum B, Green J, Nettiksimmons J, Nesper A, Patel A. Effect of a computerized decision support system on time to antibiotic initiation for severe sepsis. Vol 20, Academic Emergency Medicine. Academic Emergency Medicine 2013:S289 [FREE Full text]
  30. Narayanan N, Gross K, Pintens M, Fee C, MacDougall C. Effect of an electronic medical record alert for severe sepsis among ED patients. Am J Emerg Med 2016 Mar;34(2):185-188. [CrossRef] [Medline]
  31. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform 2010;1(4):394-407 [FREE Full text] [CrossRef] [Medline]
  32. Austrian J, Jamin C, Doty G, Blecker S. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc 2018 May 1;25(5):523-529. [CrossRef] [Medline]
  33. Pulia M, Vuong I, Schandel C, Sharp B. Impact of an Electronic Health Record Sepsis Screen on Antibiotic Stewardship in the Emergency Department. Academic Emergency Medicine 2016;23:S147.
  34. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung 2014;43(6):546-549. [CrossRef] [Medline]
  35. Mathews K, Budde J, Glasser A, Lorin S, Powell C. Impact of an in-patient electronic clinical decision support tool on sepsis-related mortality. Crit Care Med 2014;42:A1594. [CrossRef]
  36. Wheeler AP, Bernard GR. Treating patients with severe sepsis. N Engl J Med 1999 Jan 21;340(3):207-214. [CrossRef] [Medline]
  37. Makam AN, Nguyen OK, Auerbach AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med 2015 Jun;10(6):396-402 [FREE Full text] [CrossRef] [Medline]
  38. Brun-Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med 2000;26(Suppl 1):S64-S74. [CrossRef] [Medline]
  39. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the Emergency Department. Am J Emerg Med 2019 Aug;37(8):1490-1497. [CrossRef] [Medline]
  40. Machado SM, Wilson EH, Elliott JO, Jordan K. Impact of a telemedicine eICU cart on sepsis management in a community hospital emergency department. J Telemed Telecare 2018 Apr;24(3):202-208. [CrossRef] [Medline]
  41. McCoy A, Das R. Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual 2017;6(2):e000158 [FREE Full text] [CrossRef] [Medline]
  42. Westphal G, Pereira A, Fachin S, Sperotto G, Gonçalves M, Albino L, et al. An electronic warning system helps reduce the time to diagnosis of sepsis. Rev Bras Ter Intensiva 2018;30(4):414-422 [FREE Full text] [CrossRef] [Medline]
  43. Downing NL, Rolnick J, Poole SF, Hall E, Wessels AJ, Heidenreich P, et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ Qual Saf 2019 Sep;28(9):762-768 [FREE Full text] [CrossRef] [Medline]
  44. Shimabukuro DW, Barton CW, Feldman MD, Mataraso SJ, Das R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res 2017;4(1):e000234 [FREE Full text] [CrossRef] [Medline]

BRC: Biomedical Research Centre
ED: emergency department
EHR: electronic health record
ICU: intensive care unit
LOS: length of stay
NOS: Newcastle-Ottawa scale
NIHR: National Institute for Health Research
PDSA: plan-do-study-act
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-analyses
PSTRC: Patient Safety Translational Research Centre
RCT: randomized controlled trial
SBP: systolic blood pressure
SOFA: sequential organ failure assessment
SIRS: systemic inflammatory response syndrome

Edited by G Eysenbach; submitted 28.06.19; peer-reviewed by L Shallcross, S Manaktala; comments to author 10.08.19; revised version received 06.09.19; accepted 03.10.19; published 20.12.19


©Meera Joshi, Hutan Ashrafian, Sonal Arora, Sadia Khan, Graham Cooke, Ara Darzi. Originally published in the Journal of Medical Internet Research (, 20.12.2019.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.