Predicting Patient Deterioration: A Review of Tools in the Digital Hospital Setting

Introduction

Background

In recent years, proactive clinical processes have been developed to target timely and appropriate care for deteriorating or high-risk patients. Emergency responses such as Rapid Response Systems have been implemented with the aim to intervene and avoid preventable death, cardiac arrest, or transfer to an intensive care unit (ICU) in adult [1-3] and pediatric [4,5] patients. These systems have evolved to consist of a recognition (afferent) limb, commonly known as early warning scores (EWSs), and a response (efferent) limb (escalation and intervention). The responders rely on an accurate recognition limb, which in turn relies on a combination of empirical rules, statistical models, and clinical judgment to recognize deterioration. Initial EWSs were limited to vital signs, as these were the only routinely collected physiological data available for analysis in real time. The EWSs are currently available as paper or digital tools and vary significantly in their modeling, design, and escalation guidance [3,6-10].

The growth of rich, detailed, and dynamic clinical digital documentation in electronic medical records (EMRs) raises the possibility of using a broader range of clinical data, including pathology and diagnoses. An EMR collects the detailed phenotype of the patient in real time. Data collected, such as previous history, comorbidities, and demographic descriptors, are static during an admission. Observations of dynamic processes such vital signs, clinical measurements, imaging, and laboratory results that document biological and pathological processes are also recorded and continuously updated. Further evolving diagnoses, events, and interventions (eg, operations and drugs that are administered) capture the changing status of a patient. Finally, a rich source of dynamic information lies in the metadata: the timing, frequency, and location of actions and observations that occur to the patient and patient movements in the system. These data enable refined predictive models and more effective, patient-specific treatments. To create clinical decision support, these data must be analyzed and risk interpreted, and then critically, the clinical decision support communicating this real-time risk needs to be engineered back into the routine clinical workflows of the clinicians caring for the patient.

There is a diversity of models for predicting patient deterioration. Some risk estimates are static systems that identify high-risk patients at the time of diagnosis and allow triage of patients to a higher intensity care destination. Other approaches use vital sign observations to maintain an up-to-date risk evaluation. Typically, these dynamic systems either identify an extreme singular derangement or use weighted sums of a few vital signs and their variation from normal values. In both situations, the likelihood of deterioration and poor outcome anticipated with worsening values is increased. In some cases, these models have been developed and validated only for specific and narrow patient groups [11-13], whereas other general models are applicable to wider adult or pediatric ward patients [3-5,10,14].

Objectives

The aim of this review is to identify studies conducted within a general hospital setting that have attempted to develop prediction algorithms for detecting a deteriorating ward patient in real time, based primarily on routinely collected EMR data. This review includes model statistical validation and, where available, the results of digital hospital implementation of new or existing prediction models or rule-based systems for predicting patient deterioration. A secondary aim is to review those successful examples for common data elements, approaches, and statistical or machine learning techniques that were associated with successful clinical use.

Methods

This review followed the recommendations of the Center for Reviews and Dissemination [15] and PRISMA (Preferred Reporting in Systematic Reviews and Meta-Analyses) [16].

Data Sources and Search Strategy

A systematic search of PubMed, Scopus, Web of Science, IEEE Xplore, and ACM Digital Library using a combination of controlled vocabulary (eg, MeSH [Medical Subject Headings] terms) and free text keywords was conducted. The search strategy was first developed for PubMed, guided by the recommendations of Hausner et al [17] and Fatehi et al [18], and transposed to other databases. Across databases, free text keywords remained the same, but controlled vocabulary was mapped where possible (eg, from MeSH to Index Terms). The search was limited by language (English), date of publication (January 1, 2008, to June 30, 2020) and type of publication (original papers).

Screening and Study Selection

Reports that have developed prediction algorithms for detection of clinical patient deterioration in real time, primarily based on routinely collected patient data, were identified. The included studies had to (1) use any kind of (electronic) patient record, (2) use an early warning tool for patient deterioration, (3) use of a system that was dynamic, or observations of a patient over time, and (4) document the model statistical accuracy and performance. The studies were peer reviewed and published in journals or conference proceedings. The focus of this review was on tools in a general hospital ward and in a real time setting. Excluded studies were those conducted solely in a critical care or an emergency department setting, those limited to a single diagnosis or organ dysfunction patient cohort, those that used a static time point or an observation to assess risk of deterioration, and those that were qualitative with no quantitative assessments.

The results of electronic database searches were exported into an EndNote [19] library, and duplicates were removed. Title and abstract screening was coordinated on Rayyan web application (Qatar Computing Research Institute) [20]. Two independent reviewers screened titles and abstracts for relevance, and unresolved differences were adjudicated by a third reviewer. Full texts of remaining papers were assessed by 2 independent reviewers against the inclusion and exclusion criteria. Unresolved differences were adjudicated by a third reviewer. Where more than one publication was found for the same project, the papers were grouped, and the publication with the most comprehensive findings was included.

Data Extraction and Synthesis

An electronic web form was developed according to the aims of this review and was used for data extraction from full-text papers. Data extraction was conducted by one reviewer and checked by a second reviewer. Owing to the heterogeneity of the included studies in terms of study designs, aims, and outcome measures, it was not possible to conduct a meta-analysis. Thus, a narrative approach was adopted for data synthesis. An assessment of study quality, including risks of bias, was conducted using PROBAST (Prediction Model Risk of Bias Assessment Tool; UMC Utrecht) [21] and ROBINS-I (Risk of Bias in Non-Randomised Intervention Studies; Cochrane) tool [22].

Results

Overview

The initial electronic search of five databases yielded 2624 records, and an additional 31 records were identified from other sources (Figure 1). After removing duplicates, 1843 unique records remained for eligibility assessment. Following the screening of titles and abstracts, 130 papers were deemed relevant, and their full texts were obtained. After a full-text inspection, 46 papers met the eligibility criteria and were included in this review.

‎View this figure

Characteristics of the Studies

Of the 46 papers in this review, the majority (37/46, 80%) were retrospective studies that attempted to develop and/or evaluate the performance of a prediction model or compare the performance of a number of predictive models using historical patient data [23-59]. One study reported both the retrospective development of a prediction model and a prospective observational study of the developed model [60]. The remaining studies included one randomized controlled trial [61], 4 before-after (implementation) studies [62-65], and 3 prospective observational studies [66-68].

Overview of Studies

Studies from the same setting and institution were grouped and ordered according to the chronological model development, evaluation, and implementation, if available.

Of the identified studies, 6 described a body of research undertaken by researchers from Kaiser Permanente in California, as well as the University of Chicago [34,35,40,44,62,63] (Table 1). Kaiser Permanente is a large integrated managed care organization in the United States and began the deployment of EMRs in its hospitals in 2006. Escobar et al [35] reported the development of the early detection of impending physiological deterioration algorithm, which was further developed and subsequently named the Advanced Alert Monitor [44] and the early detection of impending physiological deterioration version 2 algorithm [40]. These models were developed to predict the risk of unplanned ICU transfer or mortality, using historical data of hundreds of thousands of patients retrieved from the Epic system, have also been retrospectively tested in a simulation study for feasibility [40], piloted in 2 hospitals [62], and more recently implemented as the Advanced Alert Monitor program and evaluated across the remaining Kaiser Permanente Hospitals [63,69,70].

Another group of papers from the University of Chicago program was based on experiments using the electronic patient records from a number of hospitals in the Chicago area [29-33,37,66] (Table 2). Churpek et al [29] in several papers reported the development and validation of the electronic Cardiac Arrest Risk Triage score and other tools for predicting cardiac arrest and ICU transfer in Chicago (Table 2). They also showed that mortality and cardiac arrest were easier to predict than ICU transfer [29], and machine learning methods were more accurate than logistic regression for predicting patient deterioration [32].

Several other studies have evaluated screening tools such as the National Early Warning Score, Modified Early Warning Score, Rothman Index, and Sequential Organ Failure Assessment, for the early warning and detection of patient deterioration in hospital settings across different countries [26,39,41,50,52,54,57,59,64,65,67] (Table 3). The evaluation included applying tools retrospectively on historical clinical data to assess the feasibility of future use as well as assessing tools prospectively alongside the standard clinical systems. The studies of screening tools reported differing results of both good and poor predictive accuracy and usefulness for escalation of care through medical emergency team (MET) or rapid response team (Table 3).

Of the studies detailing the implementation of scoring tools across institutions [50-52,61,62,64,65], high-risk patients were appropriately identified as aiding in clinical response (Table 4). However, when comparing intervention and control patient cohorts, differing results were seen with either significant reductions or no impact on the assessed deterioration events.

Within the reviewed papers, deterioration prediction methodologies included single and multiparameter scoring tools, such as the National Early Warning Score and Modified Early Warning Score, as well as statistical and machine learning methods. Single and multiparameter scores were derived from a set of vital sign threshold derangements (Tables 3 and 4). The statistical and machine learning methods included logistic regression, survival models, Cox regression, Gaussian process regression, Markov models, decision trees (random forest and gradient boosted trees), K-nearest neighbor, support vector machine, and neural networks (Tables 1 and 2, and Multimedia Appendix 1 [23,24,27,28,36,38,42,43,45-47,49,51,53,55,58, 60,68]). Most of these models attempted to account for changes in physiologic measures over time using novel model frameworks, for example, taking a sliding time window looking forward or backward in time to predict outcomes.

Across all the studies, models of greater complexity and statistical and machine learning methods were shown to have superior performance in predicting deterioration than scoring tools (Multimedia Appendix 1). For example, there was timelier detection and earlier warning of high deterioration risk in more complex models. Furthermore, the discrimination of patient deterioration using statistical and machine learning methods (as assessed with the area under the receiver operating characteristic curve or c statistic) outperformed conventional tools.

All patient deterioration prediction tools used vital sign measures, most commonly blood pressure, heart rate, respiratory rate, temperature, oxygen saturation, and a level of consciousness measure (Multimedia Appendix 2, Tables S2 and S3 [23-39,41-45,47-61,63-65,67,68]). In addition, most of the models included basic patient demographic data, such as age and gender, as well as administrative measures, such as admission status, time since admission, length of stay, and patient location. Many of the models attempted to incorporate various pathology or laboratory test results where available, noting that they would experience some level of time delay. Composite indices or scores for severity of illness, longitudinal comorbidities, and combined laboratory results were often included in models where available. The more complex models considered higher-level features of their data such as the frequency, change, minimum, maximum, moving average, and patterns of physiological measures over time [23,24,38,42,45,56,58].

Of the studies that reported missing data, the majority were filled by propagating a previous value carried forward if a current value within a set time window was not measured. If no prior value was available, values were imputed with a representative value such as a population-based estimate or normal value. In models that consider dynamic irregularly sampled physiologic time series data, Gaussian process models were used to deal with sparsity in the data [23,24,55].

Overall, the quality of the studies included in this review was high, with low and unclear risks of bias and concern for the applicability of prediction models in addressing our review questions (Multimedia Appendix 2, Table S2). The majority of prediction model studies appropriately selected model data for inclusion, assessed model predictors and outcomes, approached model development, and adequately tested models. Studies assessed as unclear were because of ambiguous details in reporting the number of participants or samples per derivation and validation data sets, handling of missing data, and lack of detail in model validation. The study quality of the implementation studies we identified was also high, with low to moderate risk of bias identified (Multimedia Appendix 2, Table S4). Moderate concerns were because of the lack of adjustment for potential confounding, participant selection, and lack of detail for handling of missing data.

Discussion

Principal Findings

This review examines the use of data sets collected in EMRs to develop and implement decision support for clinicians to predict and prevent inpatient deterioration. The current literature confirms that it is possible for routinely collected EMR data to be used to anticipate patient deterioration. However, there are few reports on the performance and efficacy of these systems when used in clinical settings. Despite the wide and increasing adoption of EMRs, the successful implementation of EMR-based early warning systems or their impact on patient-centered outcomes is not commonly reported.

The studies that met the eligibility criteria and the variability in methods created a narrative review rather than a quantitative review. There was considerable variation in the models developed, methodological approach, and data collected. The common study designs (retrospective, small cohorts, and before and after studies) pose a risk of bias. The institutions where models were developed and implemented predominately from the United States, which may pose additional risks of bias because of the nature of the health care system and environment. The heterogeneity in the methodology of the developed models makes comparison of their statistical accuracy and performance difficult. The area under the receiver operating characteristic curve statistic has been used as a comparison metric, but it is important to note the differences in model specification, complexity, and outcome when interpreting model output (eg, scores) and their usefulness for implementation in a real-time clinical setting. In addition, the reporting of superior performance in statistics from models of greater complexity may be the result of model overfitting if this has not been appropriately addressed in model development. Consideration should be made such that data used to develop early warning tools reflect previous cohorts of patients and may not be relevant for future cohorts. It is reasonable that retrospective patterns of deterioration will repeat themselves in the future; however, with advances in health care technology and the potential for new and emerging health disease trends, adapting or updating tools will be required to maintain relevance. We conclude that the effectiveness of current EMR-based digital early warning tools remains promising but has not been reproducibly demonstrated.

We propose that there are multiple factors that make a patient-centered outcome evaluation of EMR-based deterioration prediction difficult. Demonstration of improved patient outcomes following early warning tool implementation relies on the successful performance of all four components of implementation, recognition, escalation, and response. Where there has been successful statistical validation for the EWSs, validation of the escalation and response (ie, implementation) has been elusive. Several factors may contribute, which are not unique to digital warning implementation. The current metrics for outcome measures are problematic. Death, an outcome that is easy to count, does not always account for treatment limitations. Cardiac arrest events are rare; therefore, they are insensitive measures. ICU transfer can reflect MET resourcing deficiencies, ICU bed availability, and local admission practices or end-of-life planning rather than preventable deterioration. Emergency callout rates may reflect individual clinician sentiment rather than true risk or deterioration, or alternatively, not be called when they should be. There is inconsistency in efferent limb performance (assessment, intervention, transfer, monitoring, and follow-up). The time required for implementation and translation into improved outcomes will frequently be confounded by other system improvements, changes within an individual health service, or the maturity of the rapid response system itself. Therefore, the relationship between implementation and patient outcomes remains unstudied, rather than unproven by the current studies.

These issues will continue to challenge the empirical validation of a complete rapid response system. The lifecycle of these technologies has yet to mature to the point where evaluation and assessment of a possible effective intervention can be performed. One example of practical limitations is to recognize which data are contemporaneously available. Data can only inform a ward-based, real-time prediction model once the information is available in the EMR for analysis. This is relevant to model development and evaluation. For a pathology result, for example, there are times of sample collection, arrival at the laboratory or analysis, and the time the result is available for integration into the clinical picture. The results of the test are only available for integration once they are available in the clinical space. However, the metadata confirming that the sample was actually ordered or collected, where it was done, and the number of samples, is immediately available. The result of a positive blood culture can take hours to days, and a definitive negative result can only be finalized 2-5 days after collection. Providing actionable decision support with such dynamic data sets is a challenge, and it could be argued that the performance of the model on an experimental data set is irrelevant unless, once deployed to the EMR, the relevant decision support is actionable by clinicians in time to avoid an adverse outcome.

As maturity grows in the development cycle within EMRs, there will be potential opportunities to improve EMR-based deterioration tools and assess their impact on patient care. The development of algorithms can help monitor in real time rich clinical data from our patients, including vital signs, investigation results, drug prescription, and provide useful decision support to improve care trajectories. These clinical data can then be used at the patient level to provide visibility of deteriorating or at-risk patients to individual clinicians, wards, clinical teams, and the MET responders, and place decision support back into the EMR to support clinicians in accurately predicting and preventing inpatient deterioration. At the system level, the data could provide feedback to guide the optimization of the digital or clinician interaction and maximize response efficiency. Successful implementation should improve patient-centered outcomes, reduce suffering, incapacity, and mortality, as well as reduce length of stay, increase hospital capacity, improve efficiency, and increase care delivery to meet ever-increasing demands.

One final, but highly important, factor for consideration in the future implementation of EMR-based deterioration tools is the aspect of medical device regulation. For example, with the recent introduction of regulations for software based medical devices in Australia by the Therapeutic Goods Administration, clinical decision support software such as algorithms to predict patient deterioration that meet the definition of a medical device would be subject to regulation and possibly inclusion in the Australian Register of Therapeutic Goods. The legislation also allows for clinical decision support software to be considered as an exempt medical device subject to how it is intended to be used. Either way, it is expected that regulation will lead to increased clinical acceptance and uptake of such algorithms.

Conclusions

The development and accuracy of digital EWSs is increasing, facilitated by the growing availability of digital data sets. However, despite the relative performance of algorithms that can predict patient deterioration, the current literature shows that limited deployment of such algorithms into clinical practice is associated with improvement in patient outcomes. There is a paucity of quality studies in this area, and further work is needed to explore potential clinical benefits, including optimizing of the digital or clinician interaction, consideration of limitations in implementation, such as the requirement for real-time data availability, and use of standardized measures.