Acute kidney injury (AKI)—a sudden decline in kidney function—can be caused by hypovolemia, infection (including severe sepsis), nephrotoxicity, primary renal diseases, and urinary tract obstruction [1]. Affecting more than 18% of hospitalized patients [2], it is associated with prolonged hospital stay, need for acute renal replacement therapy (RRT), or intensive care admission as well as the development of chronic kidney disease and the need for long-term dialysis [3-5]. Although AKI may be a marker of systemic physiological decompensation in acute illnesses (eg, sepsis, trauma, or high-risk surgery), AKI itself might directly cause additional deaths through, for instance, metabolic derangement or extracellular fluid volume overload [6]. Such impacts are expensive; AKI confers excess annual costs of £1 billion to the English National Health Service (NHS) [7]. Similar excess health care costs have been demonstrated in other health systems [8].

AKI management involves the identification and treatment of life-threatening complications and medical or surgical treatment of underlying cause, supportive care (including RRT, where necessary), and interventions to reduce risk of recurrence [9]. In response to poor outcomes and variations in care delivery [10], automated AKI alerts (using standardized definitions of its presence and severity based on increases in serum creatinine [11]) have been delivered using messages in electronic health record systems or through hospital pagers [12]. In England, this approach has been applied through the embedding of an AKI detection algorithm—The NHS Early Detection Algorithm, NHSEDA (Multimedia Appendix 1)—in laboratory information management systems [13]. However, evidence of the impact of electronic alerts in improving clinical outcomes in AKI is conflicting [14]. The greatest indications of improvement seem to occur when such detection systems are coupled with structured education and clinical intervention packages [15,16]. However, the delivery of such care pathways is challenging: AKI is common and of heterogeneous etiology; it presents in diverse settings; and it is normally, at least in its early stages, managed by a range of nonspecialist teams.

To address these issues, we developed a digitally enabled care pathway for AKI patients [17]. This uses a mobile app (Streams), which alerts a specialist response team to the presence of AKI in real time, simultaneously providing relevant clinical data in a user-friendly format and allowing communication of key triage decisions among team members. Members of the response team review patients using a care protocol that maps to best practice guidance [9].

We have reported the clinical impact of this digitally enabled care pathway on patients with AKI at the point of presentation to the emergency department (ED) [18]. The limited impacts we identified in this setting might reflect the difficulty of mitigating harm when AKI is well established or in the context of pathogeneses specific to community-acquired AKI. In this paper, we assess the impact of the care pathway on clinical outcomes for patients who develop AKI following hospital admission and on health care costs.

The digital pathway was implemented at the Royal Free Hospital (RFH), a large (839 beds including a 34-bed intensive treatment unit [ITU]) hospital in north London, United Kingdom. It provides acute and emergency care as well as a range of specialist, regional inpatient services (eg, hepatology, HIV and infectious disease, amyloidosis, and vascular surgery) and has a large inpatient nephrology and renal transplant service.

For the purposes of our evaluation, we used a comparator site managed by the same health care provider organization (Royal Free London NHS Foundation Trust, RFLFT) in which the intervention was not implemented. Barnet General Hospital (BGH) is an acute general hospital with 459 beds. It has a 21-bed ITU that can provide acute RRT and a liaison nephrology service. Tertiary, specialist services are not provided on this site. A number of parallel improvement initiatives were ongoing at the comparator site during the study period, including a sepsis improvement project and an active deteriorating patients improvement program.

Blood tests, including serum creatinine, are routinely undertaken on hospitalized inpatients across all wards as directed by the treating clinicians. Historically, at both sites, blood tests would be reviewed in batches by the clinicians who ordered them. Results suggesting AKI would be telephoned to relevant wards by laboratory staff. Referral for nephrology assessment would be undertaken at the discretion of the clinical teams and using hospital pagers and phones. Cases would be prioritized and treated by the nephrology teams through assessment of referral information and results on desktop computers and through bedside review. The Patient at Risk and Resuscitation Team (PARRT) provides support to ward teams for patients deemed at risk of deterioration or who trigger existing, physiology-based early warning systems.

The digitally enabled AKI care pathway and the technical architecture of the Streams app have been described in detail previously [17], and the pre-existing and novel care pathways are shown in Multimedia Appendix 1. Members of the response team undertook training before implementation. Following implementation, Streams continuously applied the NHSEDA to creatinine results for all inpatients. Using iPhones (Apple Inc), the nephrology team was alerted to all potential cases of AKI, with their AKI stage and whether metabolic complications such as hyperkalemia were present. A curated dataset was provided, which included patients’ demographic characteristics, previously coded diagnoses, and relevant results. Filters excluded children, critical care and chronic dialysis patients, and those already under the care of the nephrology inpatient team from producing alerts. Cases were triaged in-app and, when clinical review was warranted, a best-practice care protocol was delivered (Multimedia Appendix 1). This was annotated and entered into the patient’s notes alongside an advisory sticker for key nursing actions (Multimedia Appendix 1). Recovery could be monitored in-app, and repeat AKI alerts were sent if AKI had not recovered after 48 hours or if AKI severity stage increased. Nephrology members received all AKI alerts, and PARRT members received alerts only for patients with stages 2 and 3 AKI. All team members could communicate in-app; triage responses and the outcome of clinical reviews were visible in the app to other team members. Implementation of the care pathway at the RFH used existing RFH PARRT and nephrology staff and did not result in expansion in staff numbers. A diagram outlining the pre- and postintervention care pathways is provided in Multimedia Appendix 1.

At both sites, data from RFLFT hospital databases and those supporting Streams app relating to the intervention period (May to September 2017) were compared with those from a predeployment phase (May 2016 to January 2017). Data relating to patients in whom an AKI alert was generated on presentation to the hospital ED are reported elsewhere [18], and such patients were excluded from the analyses reported here.

Data collected and their sources are detailed in Table 1. The time frame to alert viewing was determined using data recorded by the Streams app. The presence of individual comorbidities and overall patient-specific Charlson comorbidity index score (which categorizes comorbidities based on the International Classification of Diseases diagnosis codes) were derived as per the method by Thygesen et al [19]. Patients were sorted into national quintiles of deprivation (quintile 1, least; quintile 5, most deprived) using Indices of Multiple Deprivation (IMD)—a measure combining 7 domains (income, employment, living environment, health, education skills and training deprivation and disability, barriers to housing and services, and crime) into a single deprivation score for a small area—by cross-referencing patients’ postcodes with the UK Government’s Indices of Deprivation 2015 dataset [20].

For the economic analysis, we used Payment Level Information and Costing System (PLICS) data supplied by the RFLFT. PLICS is a clinical costing system where costs are derived for each patient spell (ie, admission) by tracing resources used by an individual patient in diagnosis and treatment and calculating the expenditure on those resources using the actual costs incurred by the provider. PLICS has the advantage of including staffing costs and infrastructure absorbed costs. In our study, the PLICS data for hospitalized patients with AKI included the following components: total length of stay (including the length of stay in intensive care unit), pathology and radiology examinations, total theater time, theater cutting time, inpatient dialysis, and overhead costs. These data were analyzed at the spell level. We also obtained data on the costs associated with selected individual components of a spell, which we analyzed separately (ie, length of stay, pathology and radiology examinations, theater total time, and theater cutting time). However, individual cost components were based on tariffs and not fully absorbed costs. Furthermore, we could not obtain individual costs of inpatient dialysis. The final dataset used in the economic analysis comprised total and component-specific spell-level costs at the RFH and BGH, before and after the digitally enabled care pathway was introduced at the RFH.

The primary outcome was recovery of renal function (return to a serum creatinine concentration within 120% of the baseline, as defined by the NHSEDA) before hospital discharge. Table 1 describes the predefined secondary endpoints. At both sites, NHSEDA was used to identify potential AKI cases. Because the NHSEDA can produce false positives [22], 2 authors (AC and CL) clinically validated all AKI alerts produced from all periods at both hospital sites. Only clinician-confirmed episodes of AKI were included in the analysis. In this paper, we report the outcomes of inpatients producing AKI alerts outside of the ED during the predeployment and deployment phases (Figure 1). The impact of the care pathway on cardiac arrests rate was measured on a hospital level, as it was not possible to ascertain which cardiac arrests occurred among patients with AKI.

All data were pseudonymized before transfer to the University College London (UCL) for analysis. Analyses were performed using R, version 3.4.3 (R core team) [23], and Stata MP version 14 (StataCorp) [24]. Segmented regression analysis estimated the intervention effect on the primary outcome (return to a serum creatinine concentration within 120% of the baseline, as defined by the NHSEDA) and 5 secondary outcome measures: mortality within 30 days of alert, progression of AKI stage, transfer to renal/intensive care units during admission, re-admission within 30 days of discharge, and dependence on RRT 30 days after discharge. Outcomes were measured as weekly proportions. We used binomial regression models with a logit link predicting the weekly rate of each outcome. Codes 1 and 0 were applied to the period after and before the intervention, respectively. The intervention and comparator sites were coded 1 and 0, respectively. The variable time denoted the week number (1 denoting the first week of the intervention period, and negative numbers denoting weeks in the preintervention period). The statistical model used was:

where the proportion of interest is denoted by y, the variables intervention, time, and site by int, time, and site, respectively (as defined previously), and the coefficients to be estimated by β₀,...,. In addition, 2 coefficients evaluated the evidence for the intervention causing a step change in outcome: the effect of intervention estimates the step change in outcome at the start of the intervention period at the RFH. The interaction site×intervention estimates the difference-in-differences (DID) in the step change between the intervention and comparator hospital sites. We also evaluated evidence for a change in temporal trend in the outcome because of the intervention: the time×intervention interaction estimates the difference in temporal outcome trend between the intervention and preintervention periods at the RFL; the 3-way time×site×intervention interaction estimates the DID in the trend between the intervention and comparator sites.

For all models, we inspected the autocorrelation function (up to lag 15). No significant autocorrelation was detected in any model. At the point of protocol publication, it was not anticipated that we would be able to collect patient-level data relating to sociodemographic characteristics and comorbid disease.

To examine the robustness of our primary outcome analysis, we used binary logistic regression to perform a sensitivity analysis: the same model mentioned previously was used, except that (1) the outcome was defined at the patient level and (2) patient-level characteristics (age, sex, ethnicity category, index of multiple deprivation, AKI alert level, the presence of complications at the time of alert, and the presence of individual Charlson score comorbidities) were included as covariates to adjust for any differences in casemix between sites and within sites over time.

The Wilcoxon rank-sum test was used to analyze the time to creatinine recovery (where this occurred by hospital discharge). To allow for the effects of in-hospital death on this outcome, the effect of the intervention on the length of hospital stay was estimated by competing risk analysis [25]. To determine the effect of the intervention on the time to recognition of AKI, a survival analysis was performed. The Wilcoxon rank-sum and chi-square tests were used to analyze sociodemographic variables as appropriate.

A total of 500 alerts were selected randomly from all periods, and all sites were reviewed a second time to assess the reliability of case validation. Intra- and interrater reliability was determined using Cohen’s kappa coefficient (Multimedia Appendix 1).

The number of cardiac arrests was recorded monthly at both hospital sites. Data relating to those which occurred in the hospitals’ ED, cardiac catheterization laboratory, intensive care unit, coronary care unit or in patients who had a formal not for resuscitation order signed were not included in the monthly counts recorded at the hospital level. Poisson regression models with a log link and an offset variable adjusting for the number of admissions per month were used to estimate the intervention effect on this outcome. As data were collected monthly, there was a relative paucity of postintervention data points so that estimating the effect of the intervention on outcome trend was not possible. The statistical model was:

Economic analyses used generalized linear models (GLMs) to estimate DID, where costs were defined at the spell level, and patient-level characteristics (age, sex, ethnicity category, IMD, the presence of complications at the time of alert, and the presence of individual Charlson score comorbidities, such as diabetes mellitus or congestive cardiac failure) were included as covariates so as to allow adjustment for any differences in casemix between sites and within sites over time. A GLM was specified using a gamma family and log link to account for data skewness. The model used was:

where time was defined in relation to the intervention. May to September 2016 was considered preintervention (t₁), and May to September 2017 was considered postintervention (t₃). For robustness checks, we also carried out a secondary analysis, where the preintervention period was May 2016 to January 2017. The coefficient β₉ is the coefficient of interest, measuring the between-site DID, comparing the change over time at the RFH to the change over time at the BGH. We present predictive margins showing adjusted mean costs per spell at the RFH and BGH before and after the intervention was introduced at the RFH. We adjusted for clustering at the patient level to account for the possibility that patients may have had multiple spells.

The digitally enabled care pathway constituted a new standard service at the RFH. The UCL Joint Research Office reviewed the study protocol and judged that the project fell under the remit of service evaluation as per guidance from the NHS Health Research Authority [26]. As such, no patient consent was required. The evaluation was registered with the RFH Audit Lead and Medical Director. An independent data monitoring committee (which included a patient member) reviewed all analyses before preparation for publication. A full list of committee members is provided in Multimedia Appendix 1.

Estimates from the models predicting clinical outcomes are reported in Tables 4-7, as far as they relate to the research hypotheses. All estimated model coefficients are reported in Multimedia Appendix 1.

There was a significant reduction in adjusted mean costs per spell over time at the RFH but not at the BGH (Tables 8-10). There was a significant reduction in mean costs per spell at the RFH in the postimplementation period compared with the preintervention period over and above the (nonsignificant) change seen at the BGH: the DID was −£2123 per spell (95% CI=−£4024 to −£222, P=.03). For the specified secondary analysis including all periods, the DID was −£1631 per spell (95% CI=−£3218 to −£44, P=.04). No significant differences were noted in the analyses of the cost components (Multimedia Appendix 1).

The digitally enabled care pathway for the management of AKI in a large, acute hospital with a complex casemix resulted in no significant impact on the primary outcome of renal recovery or any of the other secondary clinical outcomes measured but was associated with a significant reduction in adjusted mean costs per patient admission. We did not include the costs of providing the technology, and therefore, it is not possible to judge whether or not it would be cost saving overall. Our results suggest that the digitally enabled care pathway would be cost saving, provided provision of the technology costs less than around £1600 per patient spell. The causes of the cost savings are unclear but are likely to be multifactorial, and further research to investigate these would be useful. The most important cost components contributing to this reduction (detailed in Multimedia Appendix 1) were length of stay and theater cutting time (which might itself be expected to play a role among patients requiring surgical intervention for AKI). There was a statistically significant reduction in the need for RRT at 30 days post-AKI; however, our model was not sufficiently reliable, given the low observed event rate of this outcome. The reduction in cardiac arrest rate needs to be viewed with caution because of the large number of hypothesis tests we conducted for our 6 prespecified secondary outcomes, and because this was a hospital-wide measure, this may have been influenced by other concurrently implemented initiatives. Furthermore, cardiac arrest rates also reduced at the comparator site. It is possible that both the RFH digital pathway and BGH’s quality improvement initiative were effective to some extent through different mechanisms.

There are several possible explanations for the lack of impact on renal recovery. First, this may reflect existing high standards of AKI care before implementation: 30-day mortality for preintervention patients at RFH was 14.9% compared with 18.1% nationally [27]. It is possible that our intervention may have delivered more benefit in hospitals with worse outcomes. Second, AKI arising during inpatient admission has been shown to have worse outcomes than that arising at emergency presentation [28]. This may be because AKI arising during hospital treatment may be harder to modify. Third, AKI detection using NHSEDA depends on an elevation of serum creatinine, the detection of which may lag many hours or even days after the time of renal insult [29]. In consequence, renal injury may be less modifiable by this stage, even using a rapid system of detection such as that described. Finally, it is possible that the Streams app may have had a greater impact were it to have been implemented as part of a different care pathway—perhaps, one that involved general physicians as well as specialty care.

An explanation for the possible effect of the intervention on rates of cardiac arrest emerged from qualitative data provided in our parallel paper [30]. Here, users suggested the care pathway not only enhanced early access to specialist care for deteriorating patients but also informed treatment escalation plans. The latter included institution of ceilings of care and do not resuscitate orders with patients and relatives. Both would be expected to contribute to a reduction in the recorded unexpected cardiac arrest rate.

Our data are consistent with recent reports of the benefits of e-alerting systems for AKI for patients and the wider health system. We have reported on the impact of the digitally enabled care pathway on processes of care and clinical outcomes for patients with AKI at the point of presentation to the ED. Implementation of the digitally enabled care pathway for these patients was associated with significant improvement in the reliability of AKI recognition, a reduction in time to recognize and adjust potentially nephrotoxic medications [18]. Our qualitative analysis [30] found that care pathway improved access to patient information and expedited early specialist care. Our results concur with other research findings: a recent study from Korea [31] suggested that e-alerting for inpatients improves AKI recognition and the number of patients receiving specialist review [31]. Moreover, 2 single-site quality improvement projects combining AKI alerts with care bundles and targeted staff education also improved recognition of AKI and the quality of inpatient care [32,33]. In addition, a large multicenter sequential period analysis of an alerting system warning clinicians of the possible presence of AKI next to the display of serum creatinine results resulted in a small but sustained decrease in in-hospital mortality, dialysis use, and length of stay [34]. However, similar to our research, it is unclear which components of these pathways influenced these outcomes. A number of mixed-methods analyses of e-alerting systems for AKI are still underway; results from the qualitative segments of the AKORDD [35] and TACKLING [36] studies are awaited.

Our evaluation had a number of strengths. First, this is, to our knowledge, the first study to define the economic impact of implementing a digital innovation for AKI on health systems. Second, we clinically validated all NHSEDA AKI alerts before analysis and validated this process. Third, our inclusion of a comparator site follows best practice [37], ensuring transparency in the drawing of conclusions about the active components of our intervention.

Our evaluation also had several limitations. First, longer time frames and the inclusion of multiple intervention and comparator sites would have helped overcome the effect of differences in casemix in the pre- and postintervention period (identified in our single comparator site) and may have helped to clarify any added value of the integration of our digital innovation into the care pathway. This would also have allowed us to investigate the impact on specific patient subgroups and better understand if outcomes differed between different AKI stages. It is possible, for instance, that established severe AKI is far less responsive to intervention than the early disease. It is important that such issues are prospectively addressed in future studies. Longer time frames would also have allowed us to control for any seasonal changes in outcome, which are known to occur [38] and should be borne in mind in the design of future studies. It was not possible to collect cost data relating to the innovation of the intervention site, which should be included in any future cost-benefit analyses. Finally, although time to in-app AKI recognition and virtual review by a specialist was very rapid (median 14.0 min), comparable data from the preimplementation phase could not be collected as this process is integral to the Streams app.

The digitally enabled AKI care pathway reduced inpatient health care costs and may also help reduce hospital-wide cardiac arrest rates: this result requires reanalysis in larger, multisite studies. Growing support for greater digitalization of health systems offers the opportunity to improve the quality and safety of care and to reduce its cost. However, prospective evaluation of the clinical and cost impacts of digital innovations within the context in which they are delivered will be key in delivering maximum utility for patients and health systems.