This paper is in the following e-collection/theme issue:

Digital Health Reviews (1191) Decision Support for Health Professionals (1145) Clinical Informatics (918) Electronic Health Records (971) Clinical Information and Decision Making (1339)

Published on in Vol 24, No 2 (2022): February

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/31083, first published .
Computerized Clinical Decision Support Systems for the Early Detection of Sepsis Among Adult Inpatients: Scoping Review

Computerized Clinical Decision Support Systems for the Early Detection of Sepsis Among Adult Inpatients: Scoping Review

Computerized Clinical Decision Support Systems for the Early Detection of Sepsis Among Adult Inpatients: Scoping Review

Authors of this article:

Khalia Ackermann1 Author Orcid Image ;   Jannah Baker1 Author Orcid Image ;   Malcolm Green2 Author Orcid Image ;   Mary Fullick2 Author Orcid Image ;   Hilal Varinli3 Author Orcid Image ;   Johanna Westbrook1 Author Orcid Image ;   Ling Li1 Author Orcid Image
Article Authors Cited by (8) Tweetations (19) Metrics

    Review

    1Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, Australia

    2Clinical Excellence Commission, Sydney, Australia

    3eHealth New South Wales, Sydney, Australia

    Corresponding Author:

    Khalia Ackermann, MPH

    Centre for Health Systems and Safety Research

    Australian Institute of Health Innovation

    Level 6, Talavera Road

    Macquarie University, 2109

    Australia

    Phone: 61 2 9850 2432

    Email: khalia.ackermann@mq.edu.au


    Background: Sepsis is a significant cause of morbidity and mortality worldwide. Early detection of sepsis followed promptly by treatment initiation improves patient outcomes and saves lives. Hospitals are increasingly using computerized clinical decision support (CCDS) systems for the rapid identification of adult patients with sepsis.

    Objective: This scoping review aims to systematically describe studies reporting on the use and evaluation of CCDS systems for the early detection of adult inpatients with sepsis.

    Methods: The protocol for this scoping review was previously published. A total of 10 electronic databases (MEDLINE, Embase, CINAHL, the Cochrane database, LILACS [Latin American and Caribbean Health Sciences Literature], Scopus, Web of Science, OpenGrey, ClinicalTrials.gov, and PQDT [ProQuest Dissertations and Theses]) were comprehensively searched using terms for sepsis, CCDS, and detection to identify relevant studies. Title, abstract, and full-text screening were performed by 2 independent reviewers using predefined eligibility criteria. Data charting was performed by 1 reviewer with a second reviewer checking a random sample of studies. Any disagreements were discussed with input from a third reviewer. In this review, we present the results for adult inpatients, including studies that do not specify patient age.

    Results: A search of the electronic databases retrieved 12,139 studies following duplicate removal. We identified 124 studies for inclusion after title, abstract, full-text screening, and hand searching were complete. Nearly all studies (121/124, 97.6%) were published after 2009. Half of the studies were journal articles (65/124, 52.4%), and the remainder were conference abstracts (54/124, 43.5%) and theses (5/124, 4%). Most studies used a single cohort (54/124, 43.5%) or before-after (42/124, 33.9%) approach. Across all 124 included studies, patient outcomes were the most frequently reported outcomes (107/124, 86.3%), followed by sepsis treatment and management (75/124, 60.5%), CCDS usability (14/124, 11.3%), and cost outcomes (9/124, 7.3%). For sepsis identification, the systemic inflammatory response syndrome criteria were the most commonly used, alone (50/124, 40.3%), combined with organ dysfunction (28/124, 22.6%), or combined with other criteria (23/124, 18.5%). Over half of the CCDS systems (68/124, 54.8%) were implemented alongside other sepsis-related interventions.

    Conclusions: The current body of literature investigating the implementation of CCDS systems for the early detection of adult inpatients with sepsis is extremely diverse. There is substantial variability in study design, CCDS criteria and characteristics, and outcomes measured across the identified literature. Future research on CCDS system usability, cost, and impact on sepsis morbidity is needed.

    International Registered Report Identifier (IRRID): RR2-10.2196/24899

    J Med Internet Res 2022;24(2):e31083

    doi:10.2196/31083

    Keywords

    sepsis; early detection of disease; clinical decision support systems; patient safety; electronic health records; sepsis care pathway 


    Sepsis and Early Detection

    Sepsis, defined in 2016 as “life-threatening organ dysfunction caused by a dysregulated host response to infection,” is a leading cause of death worldwide [1]. A recent study by Rudd et al [2] estimated that 48.9 million cases of sepsis were reported in 2017, with 11 million sepsis-related deaths, representing 1 in 5 of all deaths globally [2]. Furthermore, survivors of sepsis often have a decreased quality of life, including higher rates of mortality, physical disabilities, chronic illnesses, mental health issues, and cognitive impairments [3-9].

    Prompt administration of sepsis therapies, such as intravenous antimicrobials and fluid resuscitation, is associated with better patient outcomes and lower health care–related costs [10,11]. Therefore, it is critical to detect sepsis as early as possible to ensure rapid initiation of treatment [12-14]. Unfortunately, sepsis has no diagnostic gold standard and extremely heterogenous signs and symptoms, making it difficult for clinicians to distinguish it from other acute conditions [15]. The use of sepsis identification tools, such as the Quick Sepsis-Related Organ Failure Assessment, the National Early Warning Score, and the Adult Sepsis Pathway, helps facilitate early sepsis recognition [16-18]. However, these tools typically rely on manual input of vital sign information and score calculation by clinicians. Thus, timely sepsis identification hinges on vigilant and regular patient monitoring [19]. These difficulties often result in delayed sepsis diagnosis and treatment in hospitals [19,20].

    Computerized Clinical Decision Support Systems

    The extensive implementation of data-rich electronic health records in health institutions has brought the opportunity for widespread integration of digital health care support systems [21]. In particular, the incorporation of computerized clinical decision support (CCDS) into hospital systems has the potential to assist accurate and timely early sepsis detection. CCDS systems can be designed with integrated sepsis-risk warning tools that alert clinicians to patients at risk of sepsis [13,22], reducing the physical and mental workload associated with manual patient monitoring [21].

    Over the past 10 years, CCDS technology has rapidly expanded, with two distinct approaches emerging: knowledge-based CCDS systems programmed with predefined rules derived from established clinical knowledge and adaptive CCDS systems using artificial intelligence and machine learning techniques [21,23,24]. In this scoping review, we focused on the use of knowledge-based CCDS systems in sepsis detection.

    Research Questions and Aims

    The use and implementation of sepsis CCDS systems in real-world clinical settings is a novel, rapidly expanding, and highly complex field [21,25]. In this scoping review, we systematically mapped the literature available on sepsis CCDS systems with the intention of identifying knowledge gaps and informing future research. The research question directing this review is What is the evidence base for the use of knowledge-based clinical decision support systems in hospitals for early sepsis detection and how have they been evaluated?

    More specifically, through this scoping review, we aim to (1) scope the study contexts, designs, and research methods used; (2) summarize the study outcomes investigated; and (3) map the range of CCDS system designs and implementation features, such as sepsis clinical criteria.


    Overview

    The detailed methodology for conducting this scoping review was published previously in a protocol [26]. In brief, the review was guided by the Joanna Briggs Institute Reviewer’s Manual [27], the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) [28], and the 5-stage scoping review framework proposed by Arksey and O’Malley [29]. A search for current reviews and protocols on this topic was undertaken and confirmed the absence of scoping reviews. A completed PRISMA-ScR checklist is attached in Multimedia Appendix 1 [28].

    Study Selection

    We used a broad 3-step search strategy, as outlined in our protocol [26]. An experienced librarian was consulted to help construct and refine the search. The final search strategy combined terms relating to sepsis with CCDS and detection, while excluding artificial intelligence, and was used to search MEDLINE, Embase, CINAHL, the Cochrane database, LILACS (Latin American and Caribbean Health Sciences Literature), Scopus, Web of Science, OpenGrey, ClinicalTrials.gov, and PQDT (ProQuest Dissertations and Theses Global). We restricted the search to human studies in the English language. An example of the final strategy adapted for MEDLINE can be seen in Multimedia Appendix 2. The database search was undertaken in September 2020, with no date limits applied. The reference lists of relevant systematic reviews were hand-searched to identify additional studies. Any studies identified via hand searching up until the end of data extraction (early 2021) were included. We included both peer-reviewed journal articles and gray literature (ie, conference abstracts and theses).

    Following the search, duplicates were removed as was gray literature that had been published as a peer-reviewed journal article. However, we kept studies if they reported the same methods and study cohort but examined different outcomes. Using the eligibility criteria as reported in our protocol [26], 2 reviewers (KA and JB) independently performed title, abstract, and full-text screening, with any disagreements resolved through discussion or review by a third researcher (LL). Title and abstract screening was piloted with a random selection of 25 studies by both reviewers (KA and JB). Similarly, full-text screening was piloted with a random selection of 10 studies. The 2 reviewers (KA and JB) had 100% agreement during the title and abstract screen pilot, 97.6% agreement for the full title and abstract screen, 60% agreement for the full-text screen pilot, and 77.4% agreement for the full-text screen. Hand searching was completed by 1 reviewer (KA) with identified studies confirmed by a second (JB). A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram visually illustrating this process is shown in Figure 1.

    Following screening, it was determined that the results of this review would be split over 2 papers, one investigating adult or unspecified populations and another investigating pediatric, neonatal, and maternal populations. Pediatric, neonatal, and maternal populations have remarkably different sepsis presentations and physiology compared with the general adult population [30-32]. The separation of results will allow for a more meaningful analysis. Included studies with unspecified age were assumed to likely include all patients in a general hospital setting and were grouped with adult populations. This paper reports the results of all studies investigating CCDS systems studied in adults or populations with an unspecified age.

    ‎
    Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart demonstrating the study selection process. LILACS: Latin American and Caribbean Health Science Literature; PQDT: ProQuest Dissertations and Theses Global.
    View this figure

    Data Abstraction

    The data charting form used was iteratively designed based on the study aims. The form was piloted by a single reviewer (KA) and double-checked by a second (JB). Changes to the form were made following discussion between 3 reviewers (KA, JB, and LL). Data charting was performed by 1 reviewer (KA) with ongoing consultation with the review team.

    The final data charting form included the components listed in our protocol [26], with minor adjustments as reported in Multimedia Appendix 3 [33-39]. Notably, an additional category clarity of outcome reporting was added to the form to account for the variability in outcome reporting transparency. Studies were categorized as having good clarity of outcome reporting if they specified the primary outcomes, the outcome analysis method, and the outcome measure definitions and poor clarity if the outcomes were not clearly described or there was a substantial reporting discrepancy between the methods and the results. Studies were categorized as having average clarity if they fulfilled some criteria of both good and poor.

    We accepted any definition of charted data items as specified by the studies. For example, we accepted any definition of systemic inflammatory response syndrome (SIRS), any definition of sepsis, or any cost outcomes specified for the CCDS system. We defined the usability outcome category to follow the ISO definition of usability from ISO 9241-11:2018, section 3.1.1: “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specific context of use” [33]. We required usability outcomes to be specifically investigated from the perspective of end point users (ie, clinicians). To match this definition change, usability outcomes were retrospectively categorized into the effectiveness, efficiency, or satisfaction of the CCDS system from the user’s perspective.

    Analyzing and Reporting the Results

    The results were analyzed through both a narrative review and quantitative descriptive analysis. A narrative summary of the data is presented, organized by our 3 aims. The data charted for each aim are summarized into tables using frequency counts and percentages. Graphical figures were also produced, where appropriate.

    Owing to the extensive scope of the data charted, several subgroups were collapsed into larger groups to avoid issues of small cell size and to allow for a more meaningful summary. A complete list of the smaller subgroups condensed into the larger groups, organized by table and figure, can be found in Multimedia Appendix 4. Of note, nurses were frequently reported as CCDS system responding personnel and so were grouped separately from other clinicians to better highlight this.

    Ethics

    Ethical approval or consent to participate was not required for the scoping review. The data were charted from published studies, and no individual information was included.


    Study Characteristics

    Our initial search identified 22,190 studies, with 12,139 remaining after duplicate removal. Following title, abstract, and full-text screening, 149 studies met our inclusion criteria (Figure 1). Hand searching identified 10 additional studies, resulting in a total of 159 included studies. Of these 159 studies, 124 investigated adult or unspecified populations and were included in this manuscript (Figure 1). A table detailing the main study characteristics for all 124 included studies can be found in Multimedia Appendix 5 [40-163]. In total, 52.4% (65/124) of the studies were categorized as journal articles, 43.5% (54/124) as conference abstracts, and 4% (5/124) as theses (Multimedia Appendices 4 and 5).

    Aim 1: Study Context and Design

    The context and design characteristics of the studies included in this review are presented in Table 1. Of the 124 included studies, 111 (89.5%) used purely quantitative methods to evaluate CCDS systems (Table 1). Most studies (96/124, 77.4%) used either single cohort (54/124, 43.5%) or before-after (42/124, 33.9%) study designs (Table 1). Very few studies used more robust study designs, such as randomized controlled trials (5/124, 4%), controlled studies (7/124, 5.6%), or interrupted time series (4/124, 3.2%; Table 1). None of the studies reported the use of reporting guidelines. An approximately even distribution of studies was observed across different hospital settings, such as hospital-wide, and specific settings (eg, intensive care unit [ICU], emergency department, and inpatient wards; Table 1).

    All studies but 1 (123/124, 99.2%) were published from 2009 onwards, and of the journal articles, 85% (55/65) were published in 2014 or later (Figure 2). Overall, the number of journal articles published steadily increased over time. Of the 65 journal articles, 46 (71%) reported studies conducted in the United States; 2 (3%) each in Germany, Saudi Arabia, and the United Kingdom; 1 (2%) each in Australia, Brazil, Israel, and South Korea; and 9 (14%) did not report which country they were conducted in (Multimedia Appendix 6).

    Just over half (66/124, 53.2%) of the studies specified the age of the population as adult. Within these studies, there was a reasonable variation in the actual age range provided. Almost half (29/66, 44%) reported an adult population aged ≥18 years, whereas 30% (20/66) of the studies did not specify an age range further than adult. The remaining studies reported populations using thresholds such as aged >14 (1/66, 2%), ≥14 (3/66, 5%), >16 (2/66, 3%), ≥16 (3/66, 5%), ≥19 (6/66, 9%), and ≥70 (1/66, 2%) years, with 2% (1/66) of the studies inconsistently listing multiple thresholds.

    Table 1. Study context and design.
    Study characteristicsStudies, n (%)Total (N=124), n (%)

    		Conference abstract (n=54)Journal article (n=65)Thesis (n=5)
    Method

    		Quantitative50 (92.6)56 (86.2)5 (100)111 (89.5)

    		Qualitative1 (1.9)5 (7.7)0 (0)6 (4.8)

    		Mixed methods3 (5.6)4 (6.2)0 (0)7 (5.6)
    Principal study type

    		Surveys or focus groups or heuristics1 (1.9)5 (7.7)0 (0)6 (4.8)

    		Case control1 (1.9)0 (0)0 (0)1 (0.8)

    		Single cohort30 (55.6)22 (33.8)2 (40)54 (43.5)

    		Before and after13 (24.1)27 (41.5)2 (40)42 (33.9)

    		Interrupted time series0 (0)3 (4.6)1 (20)4 (3.2)

    		Controlled study3 (5.6)4 (6.2)0 (0)7 (5.6)

    		Randomized controlled trial2 (3.7)3 (4.6)0 (0)5 (4)

    		Insufficient information to determine4 (7.4)1 (1.5)0 (0)5 (4)
    Setting

    		Hospital-widea9 (16.7)17 (26.2)1 (20)27 (21.8)

    		Intensive care unit11 (20.4)14 (21.5)1 (20)26 (21)

    		Emergency department18 (33.3)16 (24.6)2 (40)36 (29)

    		Inpatient wards11 (20.4)12 (18.5)1 (20)24 (19.4)

    		Specific ward5 (9.3)6 (9.2)0 (0)11 (8.9)
    Number of sites

    		132 (59.3)37 (56.9)2 (40)71 (57.3)

    		2-56 (11.1)16 (24.6)3 (60)25 (20.2)

    		>50 (0)6 (9.2)0 (0)6 (4.8)

    		Unspecified16 (29.6)6 (9.2)0 (0)22 (17.7)
    Age group specified?

    		Yes18 (33.3)44 (67.7)4 (80)66 (53.2)

    		No36 (66.7)21 (32.3)1 (20)58 (46.8)
    Number of participants

    		<1003 (5.6)8 (12.3)0 (0)11 (8.9)

    		101-50011 (20.4)14 (21.5)1 (20)26 (21)

    		501-10006 (11.1)4 (6.2)0 (0)10 (8.1)

    		1001-10,0009 (16.7)12 (18.5)1 (20)22 (17.7)

    		>10,00110 (18.5)15(23.1)2 (40)27 (21.8)

    		Unspecified15 (27.8)12 (18.5)1 (20)28 (22.6)
    Funding

    		Yes3 (5.6)26 (40)1 (20)30 (24.2)

    		No2 (3.7)13 (20)0 (0)15 (12.1)

    		Unspecified49 (90.7)26 (40)4 (80)79 (63.7)

    aIf the study setting was not explicitly stated, it was assumed to be hospital-wide.

    ‎
    Figure 2. Number of studies by publication type and year published. Studies published in 2020 include those until September 2020. Studies published in 2021 were found through hand searching.
    View this figure

    Aim 2: Study Outcomes

    The outcomes investigated by the included journal articles and the conference abstracts and theses are presented in Table 2 and Multimedia Appendix 7, respectively. Of the 4 predefined outcome categories, patient outcomes were reported in the highest number of studies (107/124, 86.3%; Figure 3). Sepsis treatment and management outcomes were reported in 60.5% (75/124) of the studies, CCDS system usability outcomes in 11.3% (14/124), and cost outcomes in 7.3% (9/124; Figure 3).

    Overall, only 31.5% (39/124) of the studies had good clarity in outcome reporting (Figure 4). Generally, studies had average (62/124, 50%) or poor clarity (23/124, 18.5%). Unsurprisingly, journal articles had better clarity of outcome reporting, with 40% (26/65) of the articles having good clarity, compared with 22% (13/59) of the conference abstracts or theses (Figure 4).

    In the 65 journal articles, mortality was the most frequently reported patient outcome (39/65, 60%). Overall, 35 different types of mortality measures were reported 55 times across 39 studies (Multimedia Appendix 8). Of these, in-hospital mortality was the most frequently reported (13/55, 24%; Multimedia Appendix 8). Sepsis identification, length of stay, and other patient outcomes were also frequently reported, appearing in 38% (25/65), 34% (22/65), and 35% (23/65) of the articles, respectively (Table 2; see Multimedia Appendix 4 for the expanded list of included outcomes). ICU admission was the least reported patient outcome (12/65, 18%). In the sepsis treatment and management outcome category, antibiotic-related and other were the most frequently reported outcomes in journal articles (27/65, 42% and 31/65, 48%, respectively), followed by lactate-, fluids-, and blood culture–related outcomes (17/65, 26%; 14/65, 22%; and 14/65, 22%, respectively; Table 2; see Multimedia Appendix 4 for expanded list of included outcomes). Overall, sepsis bundle or protocol compliance was the least reported outcome in journal articles (12/65, 18%).

    CCDS system usability outcomes were reported in similar numbers of journal articles, with 12% (8/65) of the journal articles reporting on the efficiency of the system, 11% (7/65) on system effectiveness, and 11% (7/65) reporting on users’ satisfaction with the system (Table 2). Among the CCDS system-related cost outcomes, cost was reported in the greatest number of journal articles (5/65, 8%), whereas cost-effectiveness or savings were reported in only 5% (3/65) of the articles (Table 2).

    Table 2. Main outcomes and outcome categories in journal articles.
    Outcome categoriesOutcome classificationa, n (%b)Total, (n=65), n (%)c

    		PrimarySecondaryNot specifiedd
    Patient outcomes

    		Mortality11 (28)9 (23)19 (49)39 (60)

    		Sepsis identification10 (40)3 (12)12 (48)25 (38)

    		Length of stay3 (14)10 (45)9 (41)22 (34)

    		Intensive care unit admission0 (0)5 (42)7 (58)12 (18)

    		Other4 (17)8 (35)11 (48)23 (35)
    Sepsis treatment and management

    		Antibiotics6 (22)9 (33)12 (44)27 (42)

    		Lactate2 (12)6 (35)9 (53)17 (26)

    		Fluids2 (14)4 (29)8 (57)14 (22)

    		Blood culture2 (14)4 (29)8 (57)14 (22)

    		Sepsis bundle or protocol compliance2 (17)4 (33)6 (50)12 (18)

    		Other5 (16)7 (23)19 (61)31 (48)
    Usability

    		Efficiency0 (0)2 (25)6 (75)8 (12)

    		Effectiveness0 (0)1 (14)6 (86)7 (11)

    		Satisfaction0 (0)3 (43)4 (57)7 (11)
    Cost

    		Cost1 (20)2 (40)2 (40)5 (8)

    		Cost-effectiveness or savings0 (0)1 (33)2 (67)3 (5)

    aSome studies reported both primary, secondary, or nonspecified outcomes within the same outcome group. To avoid double-counting these studies, secondary outcomes were not counted in favor of counting primary outcomes. Similarly, nonspecified outcomes were not counted in favor of primary or secondary outcomes. For example, a study may have the primary outcome mortality (30-day) and the secondary outcome mortality (7-day), which would both fall into the mortality outcome group. In this example the study would be counted as having mortality as the primary outcome.

    bThese percentages were calculated as row percentages, that is, using the number in the “Total” column in each row as the denominator.

    cThe percentages were calculated from the number of journal articles (n=65), not the number of total outcomes. As many journal articles reported multiple outcomes, there were more than 65 outcomes in each category, and therefore, the percentages will add up to more than 100%.

    dThe study did not specify whether the outcome was primary or secondary.

    ‎
    Figure 3. Proportion of studies reporting each outcome category.
    View this figure
    ‎
    Figure 4. Clarity of outcome reporting in the studies.
    View this figure

    Aim 3: CCDS Characteristics

    The characteristics of the CCDS systems reported in the included studies are presented in Table 3. Half (64/124, 51.6%) of the studies, most of which were journal articles (44/64, 69%), implemented homegrown CCDS systems. Of the 124 studies, only 13 (10.5%), including 10 (77%) journal articles, implemented commercial CCDS systems, of which 69% (9/13) were the St John’s Sepsis Surveillance Agent (Cerner Corporation, Kansas City, Missouri, United States; Table 3). Most included studies (95/124, 76.6%) evaluated live CCDS systems only, where the CCDS was implemented and actively sending alerts. Silent CCDS, where the system would run in real time but not send clinical alerts, were implemented by 7.3% (9/124) of studies, and 11.3% (14/124) of studies implemented both silent and live CCDS, either sequentially or concurrently.

    SIRS alone was the most frequently used CCDS clinical criteria for sepsis identification (50/124, 40.3%), followed by SIRS combined with organ dysfunction (28/124, 22.6%), and SIRS combined with other criteria (23/124, 18.5%; Table 3). In addition, a diverse range of other criteria were used by 32.3% (40/124) of studies (Multimedia Appendix 4), while 7.3% (9/124) did not specify the clinical criteria used (Table 3)

    Over half of the studies reported the implementation of CCDS systems alongside numerous other related interventions (68/124, 54.8%), such as staff education programs and antibiotic order sets (Table 3). The most common type of concurrent intervention used was clinical protocols in 41.9% (52/124) of the studies (Table 3).

    Most commonly, studies reported nurses (51/124, 41.1%) or other clinicians (37/124, 29.8%) as the main CCDS alert responding personnel (Table 3). Some studies reported on CCDS with response teams (12/124, 9.7%), study coordinators (8/124, 6.5%), or other personnel (11/124, 8.9%) responding to the alerts. Of the 124 studies, 33 (26.6%) reported the use of the electronic health record to distribute CCDS alerts, 26 (21%) the use of pagers, 15 (12.1%) the use of a patient dashboard or work list, and 8 (6.5%) the use of another form of alert delivery (Table 3).

    Table 3. Computerized clinical decision support (CCDS) characteristics.
    CCDS characteristicStudies, n (%a)Total (N=124), n (%a)

    		Conference abstract (n=54)Journal article (n=65)Thesis (n=5)
    CCDS type

    		Homegrown18 (33.3)44 (67.7)2 (40)64 (51.6)

    		Commercial3 (5.6)10 (15.4)0 (0)13 (10.5)


    		St John sepsis (Cerner Corporation, Kansas City, Missouri, United States)2 (3.7)7 (10.7)0 (0)9 (7.3)


    		PREDEC ALARM (Löser Medizintechnik GmbH, Leipzig, Germany)0 (0)1 (1.5)0 (0)1 (0.8)


    		Unspecified1 (1.9)2 (3.1)0 (0)3 (2.4)

    		Unspecified33 (61.1)11 (16.9)3 (60)47 (37.9)
    Silent or live?

    		Live41 (75.8)49 (75.4)5 (100)95 (76.6)

    		Silent5 (9.3)4 (6.2)0 (0)9 (7.3)

    		Both4 (7.4)10 (15.4)0 (0)14 (11.3)

    		Unspecified4 (7.4)2 (3.1)0 (0)6 (4.8)
    CCDS criteria

    		SIRSb24 (44.4)24 (36.9)2 (40)50 (40.3)

    		SIRS + organ dysfunction11 (20.4)17 (26.2)0 (0)28 (22.6)

    		SIRS + other9 (16.7)12 (18.5)2 (40)23 (18.5)

    		Other17 (31.5)23 (35.4)0 (0)40 (32.3)

    		Unspecified3 (5.6)5 (7.7)1 (20)9 (7.3)
    Related interventions

    		Clinical protocol16 (29.6)34 (52.3)2 (40)52 (41.9)

    		Education and staff resources8 (14.8)25 (38.5)1 (20)34 (27.4)

    		Electronic or infrastructure changes2 (3.7)6 (9.2)0 (0)8 (6.5)

    		Response or leadership team4 (7.4)17 (26.2)1 (20)22 (17.7)

    		Order sets10 (18.5)17 (26.2)0 (0)27 (21.8)

    		Feedback0 (0)10 (15.4)0 (0)10 (8.1)

    		None32 (59.3)22 (33.8)2 (40)56 (45.2)
    Responding personnel

    		Nursesc12 (22.2)38 (58.5)1 (20)51 (41.1)

    		Other clinicians11 (20.4)26 (40)0 (0)37 (29.8)

    		Response team4 (7.4)8 (12.3)0 (0)12 (9.7)

    		Study coordinator0 (0)8 (12.3)0 (0)8 (6.5)

    		Other4 (7.4)6 (9.2)1 (20)11 (8.9)

    		Unspecified27 (50)7 (10.8)3 (60)37 (29.8)
    Alert delivery

    		Electronic patient record6 (11.1)27 (41.5)0 (0)33 (26.6)

    		Pager4 (7.4)20 (30.8)2 (40)26 (21.0)

    		Patient dashboard or working list4 (7.4)10 (15.4)1 (20)15 (12.1)

    		Other1 (1.9)6 (9.2)1 (20)8 (6.5)

    		Unspecified42 (77.8)19 (29.2)2 (40)63 (50.8)

    aAs some studies reported multiple characteristics within each category, there were more than the total number of studies, and therefore, the percentages may add up to more than 100%.

    bSIRS: systemic inflammatory response syndrome.

    cAs nurses are frequently reported as CCDS system responding personnel, they were grouped separately from other clinicians.


    Principal Findings

    This review canvassed 124 studies in total, representing a comprehensive overview of current research, including an extensive body of gray literature. Over half of the included studies were journal articles (65/124, 52.4%), and nearly all studies were published in the last decade, indicating the considerable volume of recent research investigating the use of CCDS systems for early detection of adult inpatients with sepsis. Our findings demonstrate the substantial diversity of studies across all three aims: (1) the context and design of the study, (2) the type and measurement of outcomes investigated, and (3) the design and implementation of the CCDS system evaluated. We identified little research into the effects of CCDS on patient morbidity or CCDS usability and cost outcomes, highlighting key knowledge gaps in the literature. Our review also underlines the need for robust study designs, as well as improved generalizability and reporting in future studies.

    Variability Across Studies

    There is extensive heterogeneity in the current literature investigating the implementation and evaluation of CCDS systems for early sepsis detection in adult hospital patients. In particular, there was considerable diversity displayed in the chosen clinical criteria for sepsis identification across the studies included in our review. Although many studies used the SIRS criteria, alone or with adjuncts, there was a substantial range of other criteria used (Multimedia Appendix 4). This can be attributed to the extremely diverse presentations of patients with sepsis, which has led to the development of numerous different clinical scores for sepsis detection [15-18,164]. In addition, our findings demonstrated variability in the method of alert delivery, personnel who respond to alerts, and concurrent implementation of related interventions. Studies were conducted across a range of different hospital settings, including hospital-wide or specific sites, such as the emergency department or ICU (Table 1). The chosen threshold for what age participants were included in the study was also quite variable, with studies defining their adult population using cutoff points ranging from 14 to 19 years and older. Finally, our review illuminated the expansive number of outcomes used to evaluate and investigate sepsis CCDS systems. Previous systematic reviews have similarly highlighted this diversity [13,21,22,165,166]. This heterogeneity across settings, participants, CCDS system characteristics, and outcomes makes it difficult to compare studies and to make general statements regarding sepsis CCDS systems.

    This diversity can be partially attributed to the novel nature of sepsis CCDS systems and the recent emergence of the field. Our findings show a vast expansion of the literature, with three-quarters of studies published since 2014 (Figure 2). Owing to this recent rapid development of the field and the simultaneous evolution of information and communication technology in health care [21,167], there is no well-established research strategy or dogma for this specific area. Consequently, different authors have designed and executed their studies using a diverse range of variables and study design methodologies.

    This variability can also be attributed to the complexity involved in the implementation of health care interventions [168,169]. To characterize this complexity, Greenhalgh et al [170] have designed the NASSS (nonadoption, abandonment, scale-up, spread, and sustainability) framework. In the case of CCDS systems for early sepsis detection in hospitals, the 7 NASSS framework domains can be identified as follows: sepsis (the condition); the CCDS system (the technology); the commercial and health-associated value of CCDS systems (value proposition and value chain); the responding personnel (the adopters); the hospital setting (the organization); the local, state, or national health system (the wider system); and software plasticity (embedding and adaptation over time). Our findings demonstrate that these domains are extremely diverse across the included studies, presenting many variables and variable combinations, consequentially expanding the complexity involved in sepsis CCDS system implementation. As the complexity of a system has been associated with its capacity for successful and sustainable implementation [25], this heterogeneity could detrimentally impact the performance of sepsis CCDS systems. To counter this issue, Greenhalgh et al [25] highlights the importance of system usability and adaptability, suggesting that a user-centered and iterative approach is needed, centralizing the involvement of relevant providers in the implementation plan. Unfortunately, our findings indicate that few of the included studies investigated the usability of sepsis CCDS systems.

    Knowledge Gaps for Future Research

    Patient Outcomes

    Although patient outcomes were the most commonly reported outcome (Figure 3), none of the included studies directly measured the effect of CCDS systems on sepsis morbidity. Surviving sepsis is associated with cognitive impairment, higher mortality rates across the life span, physical disability, and mental health issues [3,4,6,7,9]. This not only substantially reduces the quality of life of survivors of sepsis but also presents an enormous financial burden on both patients and health care systems [5,8,171,172]. Reducing sepsis morbidity rates through CCDS use would be extremely valuable for patient health and quality of life and in mitigating personal and health care–related costs. Consequently, it is highlighted as a clear gap in the evidence base.

    Usability and Cost Outcomes

    We identified inadequate investigation of CCDS-related usability and cost outcomes, with most included studies focusing on clinical outcomes. The ability of a user to successfully operate a clinical information system is critical to the success of a system [25,173-175]. This is accentuated in the busy hospital environment, where medical providers are often time poor and carry enormous mental burdens [21,42]. Of particular concern in sepsis CCDS systems is the occurrence of alert fatigue [176,177]. Alert fatigue refers to when clinicians become desensitized to clinical alerts and consequently ignore or turn off alarm systems, potentially missing real sepsis cases [176,177]. This can have serious implications for patient outcomes. Strategies to ensure good CCDS system usability include incorporating human factor design elements, integrating CCDS system sepsis workflows into current medical emergency clinical pathways, and linking CCDS systems with existing clinical deterioration policies [42,178-180]. Only 11.3% (14/124) of the studies we investigated included usability outcomes, with only 14% (2/14) of these studies [70,93] evaluating alert fatigue. This represents a clear gap in the current literature for further research to support the successful implementation of appropriate, usable, and effective CCDS systems for early sepsis detection in hospitals.

    In addition, very few studies investigated cost outcomes of CCDS system implementation. Sepsis is an extremely expensive condition to treat [171]. It has been reported to cost more than US $20 billion annually and is listed as the most financially costly condition in US hospitals [172]. Sepsis-related costs can range from extensive hospital costs during acute treatment to high long-term treatment and rehabilitation costs in survivors of sepsis [3,171,181]. Determining the cost-effectiveness of sepsis CCDS systems would assist in establishing the financial feasibility of implementation in hospitals and support widespread implementation.

    Study Design and Generalizability

    Few studies applied robust study designs such as randomized controlled trials, interrupted time series, stepped wedge clusters, and controlled trials. Future research in this area should attempt to use more rigorous methodology to present stronger evidence.

    Approximately three-quarters of the included journal articles were conducted in the United States (Multimedia Appendix 6), limiting generalizability to other settings. A recent study demonstrated that the bulk of the sepsis burden is in countries with a low, low-middle, or middle sociodemographic index [2]. Future studies investigating the use of CCDS systems for adult sepsis inpatient identification should be encouraged to examine trends in countries outside the United States. In particular, CCDS systems should be evaluated in low- to middle-income countries when possible, given the limited availability of electronic health care technology in such regions.

    Reporting and Transparency

    A large proportion of studies did not specify important study design, CCDS system, and main outcome details (Tables 1-3). An unexpectedly high number of journal articles did not report these details nor did most conference abstracts; however, this is more understandable given word limit constraints. Of particular concern is that almost two-thirds of the included journal articles were found to have an average or poor clarity of outcome reporting (Figure 4). None of the studies included in this review published the use of reporting guidelines, likely because of many journals not specifically requiring it. Overall, we found that the quality of reporting is low and identified a need for improved reporting and transparency throughout the literature.

    Strengths and Limitations

    This scoping review comprehensively canvassed the literature investigating knowledge-based implemented CCDS systems for early sepsis detection in adult hospital patients. Its strength lies in this breadth of coverage and the wide range of study elements examined. The review followed the PRISMA-ScR expansion [28] guidelines, the Joanna Briggs Institute Reviewer’s Manual [27], and the framework presented by Arksey and O’Malley [29].

    A limitation of this scoping review is that it only included studies written in English or had English translations readily available. Furthermore, only a sample of the charted data was double-checked by a second reviewer, potentially resulting in a higher error margin. However, the data charting forms were well structured, and any issues occurring during charting were fully discussed among the research team to reach consensus.

    Conclusions

    This review highlights the extensive variability in the design, outcomes, and system characteristics in studies investigating the use of CCDS for the early detection of sepsis in adult inpatients. This heterogeneity can be largely attributed to the considerable complexity of sepsis, CCDS software, and the hospital environment. Our findings have identified clear gaps in the current literature, with few studies investigating CCDS system usability, cost, or the effects on patient morbidity. There are limited studies conducted outside the United States or with robust study designs. Our findings have illustrated frequent poor reporting of CCDS system information and study outcomes. It is critically important for future research to close these knowledge gaps, ensuring comprehensive evaluation of these rapidly emerging sepsis CCDS systems.

    Acknowledgments

    The authors would like to thank Mr Jeremy Cullis, an experienced clinical librarian, for his advice and expertise in designing the final search strategy and translating it to other databases.

    The authors received no financial support or funding for this study.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist.

    PDF File (Adobe PDF File), 507 KB
    ‎
    Multimedia Appendix 2

    Final MEDLINE search strategy.

    PDF File (Adobe PDF File), 75 KB
    ‎
    Multimedia Appendix 3

    Adjustments made to data charting form.

    PDF File (Adobe PDF File), 374 KB
    ‎
    Multimedia Appendix 4

    Definitions of groups combining multiple subgroups.

    PDF File (Adobe PDF File), 196 KB
    ‎
    Multimedia Appendix 5

    Main study characteristics.

    PDF File (Adobe PDF File), 311 KB
    ‎
    Multimedia Appendix 6

    Number of journal articles by country.

    PDF File (Adobe PDF File), 71 KB
    ‎
    Multimedia Appendix 7

    Main outcomes and outcome categories in gray literature.

    PDF File (Adobe PDF File), 209 KB
    ‎
    Multimedia Appendix 8

    Types of mortality reported in journal articles.

    PDF File (Adobe PDF File), 101 KB
    1. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). J Am Med Assoc 2016 Feb 23;315(8):801-810. [CrossRef] [Medline]
    2. Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990-2017: analysis for the Global Burden of Disease Study. Lancet 2020 Jan 18;395(10219):200-211 [FREE Full text] [CrossRef] [Medline]
    3. Iwashyna TJ, Ely EW, Smith DM, Langa KM. Long-term cognitive impairment and functional disability among survivors of severe sepsis. J Am Med Assoc 2010 Oct 27;304(16):1787-1794 [FREE Full text] [CrossRef] [Medline]
    4. Winters BD, Eberlein M, Leung J, Needham DM, Pronovost PJ, Sevransky JE. Long-term mortality and quality of life in sepsis: a systematic review. Crit Care Med 2010 May;38(5):1276-1283. [CrossRef] [Medline]
    5. Yende S, Austin S, Rhodes A, Finfer S, Opal S, Thompson T, et al. Long-term quality of life among survivors of severe sepsis: analyses of two international trials. Crit Care Med 2016 Aug;44(8):1461-1467 [FREE Full text] [CrossRef] [Medline]
    6. Davydow DS, Hough CL, Langa KM, Iwashyna TJ. Symptoms of depression in survivors of severe sepsis: a prospective cohort study of older Americans. Am J Geriatr Psychiatry 2013 Sep;21(9):887-897 [FREE Full text] [CrossRef] [Medline]
    7. Barichello T, Sayana P, Giridharan VV, Arumanayagam AS, Narendran B, Giustina AD, et al. Long-term cognitive outcomes after sepsis: a translational systematic review. Mol Neurobiol 2019 Jan;56(1):186-251. [CrossRef] [Medline]
    8. Alam N, Panday RS, Heijnen JR, van Galen LS, Kramer MH, Nanayakkara PW. Long-term health related quality of life in patients with sepsis after intensive care stay: a systematic review. Acute Med 2017;16(4):164-169. [Medline]
    9. Angus DC, van der Poll T. Severe sepsis and septic shock. N Engl J Med 2013 Aug 29;369(9):840-851. [CrossRef] [Medline]
    10. Seymour CW, Gesten F, Prescott HC, Friedrich ME, Iwashyna TJ, Phillips GS, et al. Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med 2017 Jun 08;376(23):2235-2244 [FREE Full text] [CrossRef] [Medline]
    11. Paoli CJ, Reynolds MA, Sinha M, Gitlin M, Crouser E. Epidemiology and costs of sepsis in the United States-an analysis based on timing of diagnosis and severity level. Crit Care Med 2018 Dec;46(12):1889-1897 [FREE Full text] [CrossRef] [Medline]
    12. Rhodes A, Evans LE, Alhazzani W, Levy MM, Antonelli M, Ferrer R, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Intensive Care Med 2017 Mar;43(3):304-377. [CrossRef] [Medline]
    13. Joshi M, Ashrafian H, Arora S, Khan S, Cooke G, Darzi A. Digital alerting and outcomes in patients with sepsis: systematic review and meta-analysis. J Med Internet Res 2019 Dec 20;21(12):e15166 [FREE Full text] [CrossRef] [Medline]
    14. Cecconi M, Evans L, Levy M, Rhodes A. Sepsis and septic shock. Lancet 2018 Jul 07;392(10141):75-87. [CrossRef] [Medline]
    15. McLymont N, Glover GW. Scoring systems for the characterization of sepsis and associated outcomes. Ann Transl Med 2016 Dec;4(24):527 [FREE Full text] [CrossRef] [Medline]
    16. Churpek MM, Snyder A, Han X, Sokol S, Pettit N, Howell MD, et al. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017 Apr 01;195(7):906-911 [FREE Full text] [CrossRef] [Medline]
    17. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). J Am Med Assoc 2016 Feb 23;315(8):762-774. [CrossRef] [Medline]
    18. Li L, Rathnayake K, Green M, Shetty A, Fullick M, Walter S, et al. Comparison of the quick sepsis-related organ failure assessment (qSOFA) and adult sepsis pathway in predicting adverse outcomes among adult patients on general wards: a retrospective observational cohort study. Intern Med J 2020 Jan 06. [CrossRef] [Medline]
    19. Bhattacharjee P, Edelson DP, Churpek MM. Identifying patients with sepsis on the hospital wards. Chest 2017 Apr;151(4):898-907 [FREE Full text] [CrossRef] [Medline]
    20. Marik PE. Don't miss the diagnosis of sepsis!. Crit Care 2014 Sep 27;18(5):529 [FREE Full text] [CrossRef] [Medline]
    21. Wulff A, Montag S, Marschollek M, Jack T. Clinical decision-support systems for detection of systemic inflammatory response syndrome, sepsis, and septic shock in critically ill patients: a systematic review. Methods Inf Med 2019 Dec;58(S 02):43-57 [FREE Full text] [CrossRef] [Medline]
    22. Makam AN, Nguyen OK, Auerbach AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med 2015 Jun;10(6):396-402 [FREE Full text] [Medline]
    23. Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020;3:17 [FREE Full text] [CrossRef] [Medline]
    24. Petersen C, Smith J, Freimuth RR, Goodman KW, Jackson GP, Kannry J, et al. Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper. J Am Med Inform Assoc 2021 Mar 18;28(4):677-684. [CrossRef] [Medline]
    25. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Analysing the role of complexity in explaining the fortunes of technology programmes: empirical application of the NASSS framework. BMC Med 2018 May 14;16(1):66 [FREE Full text] [CrossRef] [Medline]
    26. Li L, Ackermann K, Baker J, Westbrook J. Use and evaluation of computerized clinical decision support systems for early detection of sepsis in hospitals: protocol for a scoping review. JMIR Res Protoc 2020 Nov 20;9(11):e24899 [FREE Full text] [CrossRef] [Medline]
    27. Peters M, Godfrey C, McInerney P, Munn Z, Tricco A, Khalil H. Chapter 11: scoping reviews. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. Australia: JBI - The University of Adelaide; 2020.
    28. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018 Oct 02;169(7):467-473. [CrossRef] [Medline]
    29. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005 Feb;8(1):19-32. [CrossRef]
    30. Bonet M, Pileggi VN, Rijken MJ, Coomarasamy A, Lissauer D, Souza JP, et al. Towards a consensus definition of maternal sepsis: results of a systematic review and expert consultation. Reprod Health 2017 May 30;14(1):67 [FREE Full text] [CrossRef] [Medline]
    31. Schlapbach LJ. Paediatric sepsis. Curr Opin Infect Dis 2019 Oct;32(5):497-504. [CrossRef] [Medline]
    32. Shane AL, Sánchez PJ, Stoll BJ. Neonatal sepsis. Lancet 2017 Oct 14;390(10104):1770-1780. [CrossRef] [Medline]
    33. Ergonomics of human-system interaction - Part 11: usability: definitions and concepts. ISO standard no. 9241-11:2018(EN). The International Organization for Standardization. 2018.   URL: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en [accessed 2021-02-26]
    34. Viswanathan M, Berkman N, Dryden D, Hartling L. Assessing risk of bias and confounding in observational studies of interventions or exposures : further development of the RTI item bank. In: AHRQ Methods for Effective Health Care. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013.
    35. Ranganathan P, Aggarwal R. Study designs: Part 1 - An overview and classification. Perspect Clin Res 2018;9(4):184-186 [FREE Full text] [Medline]
    36. Aggarwal R, Ranganathan P. Study designs: Part 2 - Descriptive studies. Perspect Clin Res 2019;10(1):34-36 [FREE Full text] [CrossRef] [Medline]
    37. Ranganathan P, Aggarwal R. Study designs: Part 3 - Analytical observational studies. Perspect Clin Res 2019;10(2):91-94 [FREE Full text] [CrossRef] [Medline]
    38. Aggarwal R, Ranganathan P. Study designs: Part 4 - Interventional studies. Perspect Clin Res 2019;10(3):137-139 [FREE Full text] [CrossRef] [Medline]
    39. Aggarwal R, Ranganathan P. Study designs: Part 5 - Interventional studies (II). Perspect Clin Res 2019;10(4):183-186 [FREE Full text] [Medline]
    40. Clinical score usefulness and EHR integration (ScoreInt). ClinicalTrials.gov identifier: NCT02755025. 2017 Mar 29.   URL: https://clinicaltrials.gov/ct2/show/NCT02755025 [accessed 2020-08-17]
    41. Aakre C, Franco PM, Ferreyra M, Kitson J, Li M, Herasevich V. Prospective validation of a near real-time EHR-integrated automated SOFA score calculator. Int J Med Inform 2017 Jul;103:1-6. [CrossRef] [Medline]
    42. Aakre CA, Kitson JE, Li M, Herasevich V. Iterative user interface design for automated sequential organ failure assessment score calculator in sepsis detection. JMIR Hum Factors 2017 May 18;4(2):e14 [FREE Full text] [CrossRef] [Medline]
    43. Acquah S, Stoller M, Heller M, Krupka M, Wang C, Smith M, et al. Utilizing electronic alerts and IVC ultrasound to improve outcomes for sepsis care in an urban ED. In: Proceedings of the American Thoracic Society International Conference, ATS 2014. 2014 Presented at: American Thoracic Society International Conference, ATS 2014; May 16-21, 2014; San Diego, CA United States   URL: https://www.atsjournals.org/doi/pdf/10.1164/ajrccm-conference.2014.189.1_MeetingAbstracts.A2817
    44. Afshar M, Arain E, Ye C, Gilbert E, Xie M, Lee J, et al. Patient outcomes and cost-effectiveness of a sepsis care quality improvement program in a health system. Crit Care Med 2019 Oct;47(10):1371-1379 [FREE Full text] [CrossRef] [Medline]
    45. Alsolamy S, Al Salamah M, Al Thagafi M, Al-Dorzi HM, Marini AM, Aljerian N, et al. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak 2014 Dec 05;14:105 [FREE Full text] [CrossRef] [Medline]
    46. Amland RC, Hahn-Cover KE. Clinical decision support for early recognition of sepsis. Am J Med Qual 2016;31(2):103-110 [FREE Full text] [CrossRef] [Medline]
    47. Amland RC, Haley JM, Lyons JJ. A multidisciplinary sepsis program enabled by a two-stage clinical decision support system: factors that influence patient outcomes. Am J Med Qual 2016 Nov;31(6):501-508 [FREE Full text] [CrossRef] [Medline]
    48. Amland RC, Sutariya BB. Quick sequential [Sepsis-related] organ failure assessment (qSOFA) and St. John Sepsis Surveillance Agent to detect patients at risk of sepsis: an observational cohort study. Am J Med Qual 2018;33(1):50-57 [FREE Full text] [CrossRef] [Medline]
    49. Amland RC, Sutariya BB. An investigation of sepsis surveillance and emergency treatment on patient mortality outcomes: an observational cohort study. JAMIA Open 2018 Jul;1(1):107-114 [FREE Full text] [CrossRef] [Medline]
    50. Arabi YM, Al-Dorzi HM, Alamry A, Hijazi R, Alsolamy S, Al Salamah M, et al. The impact of a multifaceted intervention including sepsis electronic alert system and sepsis response team on the outcomes of patients with sepsis and septic shock. Ann Intensive Care 2017 Dec;7(1):57 [FREE Full text] [CrossRef] [Medline]
    51. Austrian JS, Jamin CT, Doty GR, Blecker S. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc 2018 May 01;25(5):523-529 [FREE Full text] [CrossRef] [Medline]
    52. Bansal V, Festić E, Mangi MA, Decicco NA, Reid AN, Gatch EL, et al. Early machine-human interface around sepsis severity identification: from diagnosis to improved management? Acta Med Acad 2018 May;47(1):27-38 [FREE Full text] [CrossRef] [Medline]
    53. Becker T, Herres J, Kelly J. Improved identification and management of patients with sepsis, using an electronic screening tool. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine. 2018 Presented at: 2018 Annual Meeting of the Society for Academic Emergency Medicine; May 15-18, 2018; Indianapolis, IN, United States p. S81. [CrossRef]
    54. Benson L, Hasenau S, O'Connor N, Burgermeister D. The impact of a nurse practitioner rapid response team on systemic inflammatory response syndrome outcomes. Dimens Crit Care Nurs 2014;33(3):108-115. [CrossRef] [Medline]
    55. Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform 2010;1(4):394-407 [FREE Full text] [CrossRef] [Medline]
    56. Biltoft J, Restrepo C, Volk S, Casterton T, McLeese E. Use of an electronic early warning alert to improve sepsis outcomes in a community hospital. In: Proceedings of the 42nd Critical Care Congress of the Society of Critical Care Medicine, SCCM 2013. 2012 Presented at: 42nd Critical Care Congress of the Society of Critical Care Medicine, SCCM 2013; January 19-23, 2013; San Juan Puerto Rico   URL: https:/​/journals.​lww.com/​ccmjournal/​Abstract/​2012/​12001/​483__USE_OF_AN_ELECTRONIC_EARLY_WARNING_ALERT_TO.​448.​aspx [CrossRef]
    57. Bradley P, DePolo M, Tipton J, Stacks H, Holt A. Improving sepsis survival using mews for early recognition and immediate response to patient decline. In: Proceedings of the 48th Critical Care Congress of the Society of Critical Care Medicine, SCCM 2019. 2019 Presented at: 48th Critical Care Congress of the Society of Critical Care Medicine, SCCM 2019; February 17-20, 2019; San Diego, CA, United States p. 788   URL: https:/​/journals.​lww.com/​ccmjournal/​Citation/​2019/​01001/​1626__IMPROVING_SEPSIS_SURVIVAL_USING_MEWS_FOR.​1578.​aspx [CrossRef]
    58. Brandt BN, Gartner AB, Moncure M, Cannon CM, Carlton E, Cleek C, et al. Identifying severe sepsis via electronic surveillance. Am J Med Qual 2015;30(6):559-565. [CrossRef] [Medline]
    59. Brown SM, Jones J, Kuttler KG, Keddington RK, Allen TL, Haug P. Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department. BMC Emerg Med 2016 Aug 22;16(1):31 [FREE Full text] [CrossRef] [Medline]
    60. Buck KM. Developing an early sepsis alert program. J Nurs Care Qual 2014;29(2):124-132. [CrossRef] [Medline]
    61. Carlbom D, Kelly M. SePSIS: An innovative electronic warning system for in-hospital screening of SePSIS. In: Proceeings of the Critical Care Congress 2015. 2015 Presented at: Critical Care Congress 2015; January 17-21, 2015; Phoenix, AZ United States   URL: https:/​/journals.​lww.com/​ccmjournal/​Fulltext/​2014/​12001/​966__SEPSIS__AN_INNOVATIVE_ELECTRONIC_WARNING.​933.​aspx [CrossRef]
    62. Chanas T, Volles D, Sawyer R, Mallow-Corbett S. Analysis of a new best-practice advisory on time to initiation of antibiotics in surgical intensive care unit patients with septic shock. J Intensive Care Soc 2019 Feb;20(1):34-39 [FREE Full text] [CrossRef] [Medline]
    63. Chang J, Sullivan M, Shea E, Shimabukuro D. The effectiveness of a real-time electronic alert to detect severe sepsis in an intensive care unit. In: Proceedings of the Annual Meeting of the International Anesthesia Research Society. 2015 Presented at: 2015 Annual Meeting of the International Anesthesia Research Society; March 21-24, 2015; Honolulu, HI United States p. S416. [CrossRef]
    64. Colorafi KJ, Ferrell K, D'Andrea A, Colorafi J. Influencing outcomes with automated time zero for sepsis through statistical validation and process improvement. Mhealth 2019;5:36 [FREE Full text] [CrossRef] [Medline]
    65. Comlekoglu T, Nawzadi J, Glass G, Hartka T. Utility of electronic-and provider-initiated alerts in diagnosis of sepsis in the emergency department. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2020. 2020 Presented at: Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2020; May 12-15, 2020; Denver, United States p. S210. [CrossRef]
    66. Croft CA, Moore FA, Efron PA, Marker PS, Gabrielli A, Westhoff LS, et al. Computer versus paper system for recognition and management of sepsis in surgical intensive care. J Trauma Acute Care Surg 2014 Feb;76(2):311-318. [CrossRef] [Medline]
    67. Danak V. To improve sensitivity and specificity in early detection of sepsis in patients [Dissertation]. University of California, Davis. 2014.   URL: https://www.proquest.com/docview/1665309377/abstract/403F4AE572954DA6PQ/ [accessed 2020-09-18]
    68. Downing NL, Rolnick J, Poole SF, Hall E, Wessels AJ, Heidenreich P, et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ Qual Saf 2019 Sep;28(9):762-768 [FREE Full text] [CrossRef] [Medline]
    69. Dumont L, Harding AD. Development and implementation of a sepsis program. J Emerg Nurs 2013 Nov;39(6):625-630. [CrossRef] [Medline]
    70. Dziadzko MA, Harrison AM, Tiong IC, Pickering BW, Moreno Franco P, Herasevich V. Testing modes of computerized sepsis alert notification delivery systems. BMC Med Inform Decis Mak 2016 Dec 09;16(1):156 [FREE Full text] [CrossRef] [Medline]
    71. Eck T, Kropf F, Rahmlow K, Pavlovic D, Wegner A, Guderian L, et al. The Sepsis-Tool helps to recognize severe sepsis earlier. In: Proceedings of the 4th International Congress of the German Sepsis Society “Sepsis and Multiorgan Dysfunction”. 2009 Presented at: 4th International Congress of the German Sepsis Society “Sepsis and Multiorgan Dysfunction”; September 9-12, 2009; Weimer, Germany p. 65. [CrossRef]
    72. Ehrlichman P, Trach L, Patel B, Maheshwari V, Seckel M. Sensitivity and positive predictive value of cerner EMR based sepsis recognition tool. In: Proceedings of the Critical Care Congress 2015. 2015 Presented at: Critical Care Congress 2015; January 17-21, 2015; Phoenix, AZ United States p. A1597   URL: https:/​/journals.​lww.com/​ccmjournal/​Fulltext/​2014/​12001/​983__SENSITIVITY_AND_POSITIVE_PREDICTIVE_VALUE_OF.​950.​aspx [CrossRef]
    73. Engineer R, Fertel B, Podolsky S, Gullett T, Smith E. Use of clinical decision support tools to meet compliance with CMS SEP-1 sepsis guidelines. In: Proceedings of the American College of Emergency Physicians, ACEP 2016 Research Forum. 2016 Oct Presented at: American College of Emergency Physicians, ACEP 2016 Research Forum; October 16-18, 2016; Las Vegas, NV United States p. S60. [CrossRef]
    74. Eren G, Tekdos Y, Dogan M, Acicbe O, Kaya E, Hergunsel O. Septic shock alert over SIRS criteria has an impact on outcome but needs to be revised. In: Proceedings of the 36th International Symposium on Intensive Care and Emergency Medicine. 2016 Presented at: 36th International Symposium on Intensive Care and Emergency Medicine; March 15-18, 2016; Brussels, Belgium   URL: https://cyberleninka.org/article/n/861694
    75. Faisal M, Richardson D, Scally AJ, Howes R, Beatson K, Speed K, et al. Computer-aided National Early Warning Score to predict the risk of sepsis following emergency medical admission to hospital: a model development and external validation study. Can Med Assoc J 2019 Apr 08;191(14):382-389 [FREE Full text] [CrossRef] [Medline]
    76. Falimirski M, Crews H, Broach T, Bosslet G. Early sepsis detection with an electronic medical record SIRS alert program. In: Proceedings of the 36th Annual Meeting of the Surgical Infection Society. 2016 Presented at: 36th Annual Meeting of the Surgical Infection Society; May 18-21, 2016; Palm Beach, Florida p. S30. [CrossRef]
    77. Fee C, Suess E, Maruoka A, Quon T, Meghan P, Maloney S. Effect of automated, real-time, electronic health record sirs and severe sepsis alerts on bundle compliance and mortality. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2014. 2014 Presented at: Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2014; May 13-17, 2014; Dallas, TX United States p. S102. [CrossRef]
    78. Fogerty R, Sankey C, Kenyon K, Sussman S, Acker K, Sigurdsson S, et al. Pilot of a low-resource, EHR-based protocol for sepsis monitoring, alert, and intervention. In: Proceedings of the 39th Annual Meeting of the Society of General Internal Medicine, SGIM 2016. 2016 Presented at: 39th Annual Meeting of the Society of General Internal Medicine, SGIM 2016; May 11-14, 2016; Hollywood, FL United States   URL: https:/​/shmabstracts.​org/​abstract/​pilot-of-a-low-resource-ehr-based-protocol-for-sepsis-monitoring-alert-and-intervention/​
    79. Fogerty RL, Sussman LS, Kenyon K, Li F, Sukumar N, Kliger AS, et al. Using system inflammatory response syndrome as an easy-to-implement, sustainable, and automated tool for all-cause deterioration among medical inpatients. J Patient Saf 2019 Dec;15(4):74-77. [CrossRef] [Medline]
    80. Gatewood MO, Wemple M, Greco S, Kritek PA, Durvasula R. A quality improvement project to improve early sepsis care in the emergency department. BMJ Qual Saf 2015 Dec;24(12):787-795. [CrossRef] [Medline]
    81. Gerald J, Alsip J, Hicks J, Waldrum M, Dunlap N. Using the EMR to perform continuous, automated, real-time surveillance to identify hospitalized patients at risk of sepsis. In: Poster Presentation : CHEST 2011. 2011 Oct Presented at: CHEST 2011; October 22-26, 2011; Honolulu, HI United States p. 426A. [CrossRef]
    82. Gerald J, Alsip J, Hicks J, Waldrum M, Dunlap N. Predictors of in-hospital mortality among patients identified as having severe sepsis by an automated clinical decision rule. In: Proceedings of the American Thoracic Society International Conference, ATS 2011. 2011 Presented at: American Thoracic Society International Conference, ATS 2011; May 13-18, 2011; Denver, CO, United States   URL: https://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A4696 [CrossRef]
    83. Gerald J, Alsip J, Hicks J, Waldrum M, Dunlap N. Pattern of clinical and laboratory abnormalities among hospitalized patients identified as having severe sepsis by an automated clinical decision rule. In: Proceedings of the American Thoracic Society International Conference, ATS 2011. 2011 Presented at: American Thoracic Society International Conference, ATS 2011; May 13-18, 2011; Denver, CO, United States   URL: https://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2011.183.1_MeetingAbstracts.A3860 [CrossRef]
    84. Ghanem-Zoubi NO, Vardi M, Laor A, Weber G, Bitterman H. Assessment of disease-severity scoring systems for patients with sepsis in general internal medicine departments. Crit Care 2011;15(2):R95 [FREE Full text] [CrossRef] [Medline]
    85. Giesler E, Sherwin R, Courage C, Stewart S, Fiorvento A, Powell A, et al. Use of computerized sepsis care plans improved resuscitation parameters in patients with severe sepsis and septic shock. In: Proceedings of the American College of Emergency Physicians, ACEP 2014 Research Forum. 2014 Presented at: American College of Emergency Physicians, ACEP 2014 Research Forum; October 27-28, 2014; Chicago, IL, United States p. S41. [CrossRef]
    86. Gille J, Dietz A, Taha H, Sablotzki A. A sirs-based automated alarm system for the diagnosis of sepsis after burn injury. Ann Burns Fire Disasters 2017 Sep 30;30(3):177-184 [FREE Full text] [Medline]
    87. Gowda SS, Hooper M, Weavind L, Nian H, Wheeler A. Modified systemic inflammatory response syndrome (MSIRS) score and risk of mortality in medical intensive care unit (MICU) patients. In: Proceedings of the Critical Care Congress 2012. 2012 Presented at: Critical Care Congress 2012; February 4-8, 2012; Houston, TX, United States p. 154   URL: https:/​/journals.​lww.com/​ccmjournal/​Citation/​2011/​12001/​556__MODIFIED_SYSTEMIC_INFLAMMATORY_RESPONSE.​514.​aspx
    88. Gowda SS, Nian H, Weavind L. Modified SIRS criteria in peri-operative patients admitted to surgical intensive care unit: clinical course and outcome. In: Proceedings of the Critical Care Congress 2012. 2012 Presented at: Critical Care Congress 2012; February 4-8, 2012; Houston, TX, United States p. 154   URL: https:/​/journals.​lww.com/​ccmjournal/​Citation/​2011/​12001/​557__MODIFIED_SIRS_CRITERIA_IN_PERI_OPERATIVE.​515.​aspx
    89. Grek A, Booth S, Festic E, Maniaci M, Shirazi E, Thompson K, et al. Sepsis and shock response team: impact of a multidisciplinary approach to implementing surviving sepsis campaign guidelines and surviving the process. Am J Med Qual 2017;32(5):500-507. [CrossRef] [Medline]
    90. Guidi JL, Clark K, Upton MT, Faust H, Umscheid CA, Lane-Fall MB, et al. Clinician perception of the effectiveness of an automated early warning and response system for sepsis in an academic medical center. Ann Am Thorac Soc 2015 Oct;12(10):1514-1519. [CrossRef] [Medline]
    91. Guirgis FW, Jones L, Esma R, Weiss A, McCurdy K, Ferreira J, et al. Managing sepsis: electronic recognition, rapid response teams, and standardized care save lives. J Crit Care 2017 Aug;40:296-302 [FREE Full text] [CrossRef] [Medline]
    92. Harrison AM. Development, testing, and refining the failure to rescue sepsis sniffer [Dissertation]. College of Medicine, Mayo Clinic. 2015.   URL: https://www.proquest.com/openview/d6cf865a3f8eb3c4ed3583317f9ff53b/1?pq-origsite=gscholar&cbl=18750 [accessed 2020-09-18]
    93. Harrison AM, Thongprayoon C, Aakre CA, Jeng JY, Dziadzko MA, Gajic O, et al. Comparison of methods of alert acknowledgement by critical care clinicians in the ICU setting. PeerJ 2017;5:e3083 [FREE Full text] [CrossRef] [Medline]
    94. Hayden GE, Tuuri RE, Scott R, Losek JD, Blackshaw AM, Schoenling AJ, et al. Triage sepsis alert and sepsis protocol lower times to fluids and antibiotics in the ED. Am J Emerg Med 2016 Jan;34(1):1-9 [FREE Full text] [CrossRef] [Medline]
    95. Herasevich V, Pieper MS, Pulido J, Gajic O. Enrollment into a time sensitive clinical study in the critical care setting: results from computerized septic shock sniffer implementation. J Am Med Inform Assoc 2011;18(5):639-644 [FREE Full text] [CrossRef] [Medline]
    96. Hiensch R, Poeran J, Saunders-Hao P, Adams V, Powell CA, Glasser A, et al. Impact of an electronic sepsis initiative on antibiotic use and health care facility-onset Clostridium difficile infection rates. Am J Infect Control 2017 Oct 01;45(10):1091-1100. [CrossRef] [Medline]
    97. Honeyford K, Cooke GS, Kinderlerer A, Williamson E, Gilchrist M, Holmes A, Sepsis Big Room, et al. Evaluating a digital sepsis alert in a London multisite hospital network: a natural experiment using electronic health record data. J Am Med Inform Assoc 2020 Feb 01;27(2):274-283 [FREE Full text] [CrossRef] [Medline]
    98. Hooper MH, Martin JB, Weavind LM, Semler MW, Albert DW, Deane NB, et al. Automated surveillance of modified sirs criteria is an effective tool for detection of sepsis in the medical intensive care unit. In: Proceedings of the American Thoracic Society International Conference, ATS 2010. 2010 Presented at: American Thoracic Society International Conference, ATS 2010; May 14-19, 2010; New Orleans, LA, United States. [CrossRef]
    99. Hooper MH. Modified sirs criteria for detection of sepsis in the intensive care unit. In: Proceedings of the American Thoracic Society International Conference, ATS 2011. 2011 Presented at: American Thoracic Society International Conference, ATS 2011; May 13-18, 2011; Denver, CO, United States. [CrossRef]
    100. Hooper MH, Weavind L, Wheeler AP, Martin JB, Gowda SS, Semler MW, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med 2012 Jul;40(7):2096-2101 [FREE Full text] [CrossRef] [Medline]
    101. Horton DJ, Graves KK, Kukhareva PV, Johnson SA, Cedillo M, Sanford M, et al. Modified early warning score-based clinical decision support: cost impact and clinical outcomes in sepsis. JAMIA Open 2020 Jul;3(2):261-268 [FREE Full text] [CrossRef] [Medline]
    102. Huff S, Stephens K, Whiteman K, Swanson-Biearman B, Mori C. Implementation of a vital sign alert system to improve outcomes. J Nurs Care Qual 2019;34(4):346-351. [CrossRef] [Medline]
    103. Hughes MC, Roedocker A, Ehli J, Walz D, Froehlich K, White L, et al. A quality improvement project to improve sepsis-related outcomes at an integrated healthcare system. J Healthc Qual 2019;41(6):369-375. [CrossRef] [Medline]
    104. Jones SL, Ashton CM, Kiehne L, Gigliotti E, Bell-Gordon C, Disbot M, et al. Reductions in sepsis mortality and costs after design and implementation of a nurse-based early recognition and response program. Jt Comm J Qual Patient Saf 2015 Nov;41(11):483-491 [FREE Full text] [CrossRef] [Medline]
    105. Judd WR, Stephens DM, Kennedy CA. Clinical and economic impact of a quality improvement initiative to enhance early recognition and treatment of sepsis. Ann Pharmacother 2014 Oct;48(10):1269-1275. [CrossRef] [Medline]
    106. Jukes T. Using clinical decision support to assist in identification of severe sepsis: design development and implementation of an automated sepsis prompt [Dissertation]. University of California, Davis. 2013.   URL: https://www.proquest.com/docview/1449817718? [accessed 2020-09-18]
    107. Jung AD, Baker J, Droege CA, Nomellini V, Johannigman J, Holcomb JB, et al. Sooner is better: use of a real-time automated bedside dashboard improves sepsis care. J Surg Res 2018 Nov;231:373-379. [CrossRef] [Medline]
    108. Kangas C, Iverson L, Pierce D. Sepsis screening: combining early warning scores and SIRS criteria. Clin Nurs Res 2021 Jan;30(1):42-49. [CrossRef] [Medline]
    109. Krieger P, Jackson J, Sarin R, Wang C, Acquah S, Smith M, et al. A novel electronic medical record-based screening alert successfully identifies patients with severe sepsis. In: Proceedings of the American College of Emergency Physicians, ACEP Research Forum 2013. 2013 Oct Presented at: American College of Emergency Physicians, ACEP Research Forum 2013; October 14-15, 2013; Seattle, WA, United States p. S96. [CrossRef]
    110. Kurczewski L, Sweet M, McKnight R, Halbritter K. Reduction in time to first action as a result of electronic alerts for early sepsis recognition. Crit Care Nurs Q 2015;38(2):182-187. [CrossRef] [Medline]
    111. Lawson L. Use of clinical decision algorithms on medical surgical units to assist in the identification and care of patients at risk for acquiring sepsis [Dissertation]. Northcentral University. 2017.   URL: https:/​/www.​proquest.com/​openview/​9de3653d06b2e4128ccde607b239edf2/​1?pq-origsite=gscholar&cbl=18750&diss=y [accessed 2020-09-18]
    112. Liang D, Cope J, Patel R, Holder M. Comparison of two sepsis alert systems within an academic medical center. In: Proceedings of the 47th Society of Critical Care Medicine Critical Care Congress, SCCM 2018. 2018 Presented at: 47th Society of Critical Care Medicine Critical Care Congress, SCCM 2018; February 25-28, 2018; San Antonio, TX, United States p. 472. [CrossRef]
    113. MacMillan A, Rudinsky D, Han G, Elliott JO, Jordan K. Multidisciplinary approach to improve sepsis outcomes. J Healthc Qual 2019;41(4):220-227. [CrossRef] [Medline]
    114. Mann-Salinas E, Caldwell N, Serio-Melvin M, Luellen D, Chung K, Cancio L, et al. Evaluation of burn sepsis automated alerts in an intensive care unit. In: Proceedings of the Critical Care Congress 2015. 2015 Presented at: Critical Care Congress 2015; January 17-21, 2015; Phoenix, AZ, United States p. A1596. [CrossRef]
    115. Manteuffel J, Coba V, Jaehne K. An emergency department sepsis alert is associated with decreased mortality in septic patients admitted to the intensive care unit. In: Proceedings of the American College of Emergency Physicians, ACEP Research Forum 2013. 2013 Oct Presented at: American College of Emergency Physicians, ACEP Research Forum 2013; October 14-15, 2013; Seattle, WA, United States p. S63. [CrossRef]
    116. Martin-Rico P, Valdivia-Perez A, Lacalle-Martinez J, Chorda-Ribelles J, Marco-Lattur M, Lozano-Cortell N, et al. Assessing the value of a real-time electronic screening algorithm for early detection of severe sepsis in the emergency department. Crit Care 2014 Dec 3;18(S2):P22. [CrossRef]
    117. McCabe P, Team R, Thorne J, Sarumi M, Ryan A, Rappaport H. M2SEWS early warning systems and proactive screening provides early intervention and escalates care. In: Proceedings of the 45th Critical Care Congress of the Society of Critical Care Medicine, SCCM 2015. 2015 Presented at: 45th Critical Care Congress of the Society of Critical Care Medicine, SCCM 2015; February 20-24, 2016; Orlando, FL, United States p. 276-277   URL: https:/​/journals.​lww.com/​ccmjournal/​Citation/​2015/​12001/​1100__M2SEWS_EARLY_WARNING_SYSTEMS_AND_PROACTIVE.​1103.​aspx
    118. McGrane T, Rice T, Hooper M, Gowda S, Weavind L. Electronic SIRS alert facilitating recognition of sepsis by housestaff. In: Proceedings of the 40th Critical Care Congress of the Society of Critical Care Medicine. 2010 Presented at: 40th Critical Care Congress of the Society of Critical Care Medicine; January 15-19, 2011; San Diego, CA, United States p. A170. [CrossRef]
    119. McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung 2014;43(6):546-549. [CrossRef] [Medline]
    120. Meurer WJ, Smith BL, Losman ED, Sherman D, Yaksich JD, Jared JD, et al. Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc 2009 Jan;57(1):40-45. [CrossRef] [Medline]
    121. Miller K, Kowalski R, Capan M, Wu P, Mosby D, Arnold R. Assessment of nursing response to a real-time alerting tool for sepsis: a provider survey. Am J Hosp Med 2017;1(3) [FREE Full text] [Medline]
    122. Muratore S, Coughlan A, Glover JK, Statz CL, Weinert C, Kline SE, et al. Electronically triggered sepsis alert in non-intensive care inpatients using modified systemic inflammatory response syndrome criteria: a retrospective observational study with in-depth analysis of surgical patients. Surg Infect (Larchmt) 2019;20(4):278-285. [CrossRef] [Medline]
    123. Narayanan N, Gross AK, Pintens M, Fee C, MacDougall C. Effect of an electronic medical record alert for severe sepsis among ED patients. Am J Emerg Med 2016 Feb;34(2):185-188. [CrossRef] [Medline]
    124. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 2011 May;57(5):500-504. [CrossRef] [Medline]
    125. Nelson G, Kamash T, Sherwin R. The performance of an electronic medical record-based alert to identify emergency department patients with severe sepsis and septic shock. In: Proceedings of the American College of Emergency Physicians, ACEP Research Forum 2012. 2012 Oct Presented at: American College of Emergency Physicians, ACEP Research Forum 2012; October 8-9, 2012; Denver, CO, United States p. S21. [CrossRef]
    126. Nguyen SQ, Mwakalindile E, Booth JS, Hogan V, Morgan J, Prickett CT, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ 2014;2:e343 [FREE Full text] [CrossRef] [Medline]
    127. Noonan TZ. Decision making under uncertainty in the emergency department: studying the effects of cognitive biases in the diagnosis of sepsis [Dissertation]. The University of Iowa. 2018.   URL: https://www.proquest.com/openview/56541ec710f706fd3432237557287af3/1?pq-origsite=gscholar&cbl=18750 [accessed 2020-09-18]
    128. Okafor N, Mehta A, Freeman C, Doshi P. Evaluation of a sirs alert tool as a method for early detection of sepsis in the emergency department. In: Proceedings of the American College of Emergency Physicians, ACEP 2014 Research Forum. 2014 Presented at: American College of Emergency Physicians, ACEP 2014 Research Forum; October 27-28, 2014; Chicago, IL, United States p. S56. [CrossRef]
    129. Oxman D, Oettinger G, Pugliese R, El Beyrouty C, McDaniel C, Riggio J, et al. An interdisciplinary program for improving the recognition and treatment of severe sepsis. In: Proceedings of the 43rd Critical Care Congress of the Society of Critical Care Medicine, SCCM 2014. 2013 Presented at: 43rd Critical Care Congress of the Society of Critical Care Medicine, SCCM 2014; January 9-13, 2014; San Francisco, CA, United States p. A270. [CrossRef]
    130. Pandit A, Hale K, Pottinger T, Bugov D, Lacaille S, Rebal S, et al. Analysis of an automated sepsis screening alert at a tertiary cancer center. In: Poster Presentation: CHEST 2015. 2015 Oct Presented at: CHEST 2015; October 24-28, 2015; Montreal, QC, Canada p. 342A. [CrossRef]
    131. Perlin JB, Jackson E, Hall C, Mindick A, Berberich G, Korwek K, et al. 2019 John M. Eisenberg Patient Safety and Quality Awards: SPOTting sepsis to save lives: a nationwide computer algorithm for early detection of sepsis: innovation in patient safety and quality at the national level (Eisenberg Award). Jt Comm J Qual Patient Saf 2020 Jul;46(7):381-391. [CrossRef] [Medline]
    132. Pertiwi AA, Fraczkowski D, Stogis SL, Lopez KD. Using heuristic evaluation to improve sepsis alert usability. Crit Care Nurs Clin North Am 2018 Jun;30(2):297-309. [CrossRef] [Medline]
    133. Pham T, Kupchik N, Rivera S, Robinson C, Shea N. EB44 automated sepsis screening in a progressive care unit. Critical Care Nurse 2012;32(2):27-28 [FREE Full text]
    134. Powers B, Mwangi J, Ordaz E, Weber L, patel N, Uppalapati A, et al. EMR-Based sepsis alert: improving early detection. In: Proceedings of the Critical Care Congress 2015. 2015 Presented at: Critical Care Congress 2015; January 17-21, 2015; Phoenix, AZ, United States p. 1598-1599. [CrossRef]
    135. Pulia M, Vuong I, Schandel C, Sharp B. Impact of an electronic health record sepsis screen on antibiotic stewardship in the emergency department. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2016. 2016 Presented at: Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2016; May 10-13, 2016; New Orleans, LA, United States p. S147. [CrossRef]
    136. Rauch S, Fischer M, Martin C, Sablotzki A. Automated, electronic monitoring for early detection of sepsis. In: Proceedings of the 33rd International Symposium on Intensive Care and Emergency Medicine. 2013 Presented at: 33rd International Symposium on Intensive Care and Emergency Medicine; March 19-22, 2013; Brussels, Belgium p. S191. [CrossRef]
    137. Rincon TA, Manos EL, Pierce JD. Telehealth intensive care unit nurse surveillance of sepsis. Comput Inform Nurs 2017 Sep;35(9):459-464. [CrossRef] [Medline]
    138. Salamah M, Thagafi M, Reidy M, Diknash S, Al-Qahtani S, Marini A, et al. The predictive value of an electronic alert system for severe sepsis and septic shock. In: Proceedings of the American Thoracic Society International Conference, ATS 2013. 2013 Presented at: American Thoracic Society International Conference, ATS 2013; May 17-22, 2013; Philadelphia, PA, United States   URL: https://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2013.187.1_MeetingAbstracts.A3081
    139. Sankey C, Sussman LS, Kenyon K, Li F, Sukumar N, Kliger A, et al. Pilot of a low-resource, EHR-based tool for sepsis monitoring, alert, and intervention. In: Proceedings of the 40th Annual Meeting of the Society of General Internal Medicine, SGIM 2017. 2017 Presented at: 40th Annual Meeting of the Society of General Internal Medicine, SGIM 2017; April 19-22, 2017; Washington, DC, United States   URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5391321/
    140. Santistevan J, Hamedani A, Werner E, Spranger J, Sharp B. Performance of an automated severe sepsis screening tool in the emergency department. In: Proceedings of the American College of Emergency Physicians, ACEP 2017 Research Forum. 2017 Presented at: American College of Emergency Physicians, ACEP 2017 Research Forum; October 29-31, 2017; Washington, DC, United States p. 163-164. [CrossRef]
    141. Sarumi M, Garner B, Rappaport H, McCabe P, Gutierrez S, Ryan A, et al. Implementation of a standardized and reliable system improves mortality in patients with severe sepsis. In: Proceedings of the 43rd Critical Care Congress of the Society of Critical Care Medicine, SCCM 2014. 2014 Presented at: 43rd Critical Care Congress of the Society of Critical Care Medicine, SCCM 2014; January 9-13, 2014; San Francisco, CA, United States p. A273. [CrossRef]
    142. Sawyer AM, Deal EN, Labelle AJ, Witt C, Thiel SW, Heard K, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med 2011 Mar;39(3):469-473. [CrossRef] [Medline]
    143. Seetharaman S, Wilson C, Landrum M, Qasba S, Katz M, Ladikos N, et al. Does use of electronic alerts for systemic inflammatory response syndrome (sirs) to identify patients with sepsis improve mortality? Am J Med 2019 Jul;132(7):862-868. [CrossRef] [Medline]
    144. Sharp B, Schandel C, Vuong I, Masters M, Lee A, Pothof J, et al. Implementation of an electronic ED sepsis screen and alert: effect on compliance with ED sepsis quality measures. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2016. 2016 Presented at: Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2016; May 10-13, 2016; New Orleans, LA, United States p. S96. [CrossRef]
    145. Sharp B, Schandel C, Vuong I, Masters M, Lee A, Pothof J, et al. Silent no longer: sepsis recognition by electronic screening in the emergency department. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2016. 2016 Presented at: Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2016; May 10-13, 2016; New Orleans, LA, United States p. 95-96. [CrossRef]
    146. Sherwin R, Courage C, Stewart S, Robinson D, Ying H. Performance of two clinical decision support tools to identify sepsis patients in the emergency department. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2014. 2014 Presented at: Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2014; May 13-17, 2014; Dallas, TX, United States p. S279. [CrossRef]
    147. Shetty A, Murphy M, Middleton-Rennie C, Lancuba A, Green M, Lander H, et al. Evaluation of an augmented emergency department electronic medical record-based sepsis alert. Emerg Med Australas 2021 Feb 23. [CrossRef] [Medline]
    148. Song J, Cho H, Park DW, Ahn S, Kim JY, Seok H, et al. The effect of the intelligent sepsis management system on outcomes among patients with sepsis and septic shock diagnosed according to the sepsis-3 definition in the emergency department. J Clin Med 2019 Oct 27;8(11):27 [FREE Full text] [CrossRef] [Medline]
    149. Swenson K, Ferguson J, Shieh L, Rogers A. Attitudes and perceptions of medical trainees towards an electronic medical alert system for sepsis. In: Proceedings of the American Thoracic Society International Conference, ATS 2018. 2018 Presented at: American Thoracic Society International Conference, ATS 2018; May 18-23, 2018; San Diego, CA, United States   URL: https://www.atsjournals.org/doi/abs/10.1164/ajrccm-conference.2018.197.1_MeetingAbstracts.A4504
    150. Swenson K, Rogers A, Krishnan G, Shieh L. Analysis of a potential automated sepsis alert based on quick sequential organ failure assessment (qSOFA) among hospitalized floor patients. In: Proceedings of the American Thoracic Society 2019 International Conference. 2019 Presented at: American Thoracic Society 2019 International Conference; May 17-22, 2019; Dallas, TX, United States. [CrossRef]
    151. Tafelski S, Nachtigall I, Deja M, Tamarkin A, Trefzer T, Halle E, et al. Computer-assisted decision support for changing practice in severe sepsis and septic shock. J Int Med Res 2010;38(5):1605-1616 [FREE Full text] [CrossRef] [Medline]
    152. Thompson DS, Oberteuffer R, Dorman T. Sepsis alert and diagnostic system: integrating clinical systems to enhance study coordinator efficiency. Comput Inform Nurs 2003;21(1):22-27. [CrossRef] [Medline]
    153. Tsai C, Patel K, Vincent A, Verzosa N, Norris D, Tillis W, et al. Electronic best practice advisories' effectiveness in detecting sepsis in the emergency department. Ann Emerg Med 2015 Oct;66(4):91-92. [CrossRef]
    154. Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, et al. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015 Jan;10(1):26-31 [FREE Full text] [Medline]
    155. Watanabe M, Kim B, Jeng A. St John sepsis alert correlation with management of infections and antimicrobial selection: a pilot study. In: Proceedings of the Open Forum Infectious Diseases, ID Week 2017. 2017 Presented at: Open Forum Infectious Diseases, ID Week 2017; October 4-8, 2017; San Diego, CA, United States p. S513. [CrossRef]
    156. Wawrose R, Baraniuk M, Standiford L, Wade C, Holcomb J, Moore L. Comparison of sepsis screening tools' ability to detect sepsis accurately. Surg Infect (Larchmt) 2016 Oct;17(5):525-529. [CrossRef] [Medline]
    157. Westphal GA, Pereira AB, Fachin SM, Sperotto G, Gonçalves M, Albino L, et al. An electronic warning system helps reduce the time to diagnosis of sepsis. Rev Bras Ter Intensiva 2018;30(4):414-422 [FREE Full text] [Medline]
    158. Winterbottom F. Early warning: detecting sepsis in every patient, every shift requires timely vital sign documentation. In: Proceedings of the Ochsner 13th Annual Research Day Abstracts. 2016 Presented at: Ochsner 13th Annual Research Day Abstracts; May 17, 2016; New Orleans, LA, United States p. 398   URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5024823/
    159. Wolpaw B, Kress N, Grant R, Wahl D, Garver-Hume A, Villanueva S, et al. EHR-based sepsis warning system improves detection and treatment of hospital-acquired sepsis in an acute care hospital setting. In: Proceedings of the 41st Annual Meeting of the Society of General Internal Medicine, SGIM 2018. 2018 Presented at: 41st Annual Meeting of the Society of General Internal Medicine, SGIM 2018; April 11-14, 2018; Denver, CO, United States   URL: https://link.springer.com/content/pdf/10.1007%2Fs11606-018-4413-y.pdf
    160. Yeich T, Brown S, Thomas K, Walker R. The sepsis alert: real-time electronic health record surveillance in the emergency department and sepsis outcomes. In: Proceedings of the Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2012. 2012 Presented at: Annual Meeting of the Society for Academic Emergency Medicine, SAEM 2012; May 9-12, 2012; Chicago, IL, United States p. S49. [CrossRef]
    161. Yohannes S. Multimodal sepsis performance improvement initiative improves early and appropriate treatment, reduces sepsis-related readmissions, and reduces overall mortality. In: Proceedings of the Open Forum Infectious Diseases, ID Week 2019. 2019 Presented at: Open Forum Infectious Diseases, ID Week 2019; October 2-6, 2019; Washington, DC, United States p. S343. [CrossRef]
    162. Zaragoza R, Sancho S, Hurtado C, Camarena J, González R, Borrás S, et al. SEPSIS SAVER. Preliminary results of a pilot implementation of a new automatic electronic alert for early detection of sepsis based on Sepsis 3 definition. In: Proceedings of the 32nd European Society of Intensive Care Medicine Annual Congress, ESICM 2019. 2019 Presented at: 32nd European Society of Intensive Care Medicine Annual Congress, ESICM 2019; September 28 - October 2, 2019; Berlin, Germany   URL: https://icm-experimental.springeropen.com/articles/supplements/volume-7-supplement-3
    163. Zuick S, Asch DA, Graustein A, Urbani R, Carrol B. Can a computerized sepsis screening and alert system accurately diagnose sepsis in hospitalized floor patients and potentially provide opportunities for early intervention? A pilot study. Intens Crit Care 2016;02(03):1-7. [CrossRef]
    164. Uffen JW, Oosterheert JJ, Schweitzer VA, Thursky K, Kaasjager HA, Ekkelenkamp MB. Interventions for rapid recognition and treatment of sepsis in the emergency department: a narrative review. Clin Microbiol Infect 2020 Feb 29. [CrossRef] [Medline]
    165. Despins LA. Automated detection of sepsis using electronic medical record data: a systematic review. J Healthc Qual 2016 Sep 13. [CrossRef] [Medline]
    166. Warttig S, Alderson P, Evans DJ, Lewis SR, Kourbeti IS, Smith AF. Automated monitoring compared to standard care for the early detection of sepsis in critically ill patients. Cochrane Database Syst Rev 2018 Jun 25;6:CD012404 [FREE Full text] [CrossRef] [Medline]
    167. Feldman SS, Buchalter S, Hayes LW. Health information technology in healthcare quality and patient safety: literature review. JMIR Med Inform 2018 Jun 04;6(2):e10264 [FREE Full text] [CrossRef] [Medline]
    168. Petticrew M, Knai C, Thomas J, Rehfuess EA, Noyes J, Gerhardus A, et al. Implications of a complexity perspective for systematic reviews and guideline development in health decision making. BMJ Glob Health 2019;4(Suppl 1):e000899 [FREE Full text] [CrossRef] [Medline]
    169. Plsek PE, Greenhalgh T. Complexity science: the challenge of complexity in health care. Br Med J 2001 Sep 15;323(7313):625-628 [FREE Full text] [CrossRef] [Medline]
    170. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017 Nov 01;19(11):e367 [FREE Full text] [CrossRef] [Medline]
    171. Arefian H, Heublein S, Scherag A, Brunkhorst FM, Younis MZ, Moerer O, et al. Hospital-related cost of sepsis: a systematic review. J Infect 2017 Feb;74(2):107-117. [CrossRef] [Medline]
    172. Torio C, Andrews R. National inpatient hospital costs: the most expensive conditions by payer. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013.
    173. Peute LW, Spithoven R, Bakker PJ, Jaspers MW. Usability studies on interactive health information systems; where do we stand? Stud Health Technol Inform 2008;136:327-332. [Medline]
    174. Miller K, Mosby D, Capan M, Kowalski R, Ratwani R, Noaiseh Y, et al. Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. J Am Med Inform Assoc 2018 May 01;25(5):585-592 [FREE Full text] [CrossRef] [Medline]
    175. Hardenbol AX, Knols B, Louws M, Meulendijk M, Askari M. Usability aspects of medication-related decision support systems in the outpatient setting: a systematic literature review. Health Informatics J 2020 Mar;26(1):72-87 [FREE Full text] [CrossRef] [Medline]
    176. Harrison AM, Gajic O, Pickering BW, Herasevich V. Development and implementation of sepsis alert systems. Clin Chest Med 2016 Jun;37(2):219-229 [FREE Full text] [CrossRef] [Medline]
    177. The Lancet Respiratory Medicine. Crying wolf: the growing fatigue around sepsis alerts. Lancet Respir Med 2018 Mar;6(3):161. [CrossRef] [Medline]
    178. Khairat S, Marc D, Crosby W, Al Sanousi A. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med Inform 2018 Apr 18;6(2):e24 [FREE Full text] [CrossRef] [Medline]
    179. Jones D, Duke G, Green J, Briedis J, Bellomo R, Casamento A, et al. Medical emergency team syndromes and an approach to their management. Crit Care 2006 Feb;10(1):R30 [FREE Full text] [CrossRef] [Medline]
    180. Henning DJ, Oedorf K, Day DE, Redfield CS, Huguenel CJ, Roberts JC, et al. Derivation and validation of predictive factors for clinical deterioration after admission in emergency department patients presenting with abnormal vital signs without shock. West J Emerg Med 2015 Dec;16(7):1059-1066 [FREE Full text] [CrossRef] [Medline]
    181. Hajj J, Blaine N, Salavaci J, Jacoby D. The "Centrality of Sepsis": a review on incidence, mortality, and cost of care. Healthcare (Basel) 2018 Jul 30;6(3) [FREE Full text] [CrossRef] [Medline]

    ‎
    CCDS: computerized clinical decision support
    ICU: intensive care unit
    LILACS: Latin American and Caribbean Health Sciences Literature
    PQDT: ProQuest Dissertations and Theses Global
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews
    SIRS: systemic inflammatory response syndrome

    Edited by R Kukafka; submitted 17.06.21; peer-reviewed by K Honeyford, J Walsh; comments to author 30.07.21; revised version received 23.09.21; accepted 29.10.21; published 23.02.22

    Copyright

    ©Khalia Ackermann, Jannah Baker, Malcolm Green, Mary Fullick, Hilal Varinli, Johanna Westbrook, Ling Li. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 23.02.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.