This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
One reason for the introduction of digital technologies into health care has been to try to improve safety and patient outcomes by providing real-time access to patient data and enhancing communication among health care professionals. However, the adoption of such technologies into clinical pathways has been less examined, and the impacts on users and the broader health system are poorly understood. We sought to address this by studying the impacts of introducing a digitally enabled care pathway for patients with acute kidney injury (AKI) at a tertiary referral hospital in the United Kingdom. A dedicated clinical response team—comprising existing nephrology and patient-at-risk and resuscitation teams—received AKI alerts in real time via Streams, a mobile app. Here, we present a qualitative evaluation of the experiences of users and other health care professionals whose work was affected by the implementation of the care pathway.
The aim of this study was to qualitatively evaluate the impact of mobile results viewing and automated alerting as part of a digitally enabled care pathway on the working practices of users and their interprofessional relationships.
A total of 19 semistructured interviews were conducted with members of the AKI response team and clinicians with whom they interacted across the hospital. Interviews were analyzed using inductive and deductive thematic analysis.
The digitally enabled care pathway improved access to patient information and expedited early specialist care. Opportunities were identified for more constructive planning of end-of-life care due to the earlier detection and alerting of deterioration. However, the shift toward early detection also highlighted resource constraints and some clinical uncertainty about the value of intervening at this stage. The real-time availability of information altered communication flows within and between clinical teams and across professional groups.
Digital technologies allow early detection of adverse events and of patients at risk of deterioration, with the potential to improve outcomes. They may also increase the efficiency of health care professionals’ working practices. However, when planning and implementing digital information innovations in health care, the following factors should also be considered: the provision of clinical training to effectively manage early detection, resources to cope with additional workload, support to manage perceived information overload, and the optimization of algorithms to minimize unnecessary alerts.
In many health systems, the ageing demographic of hospital patients is accompanied by worsening health and a greater need for diverse investigations and treatments. As a result, care pathways are increasingly complex and ever more reliant on access to relevant data and on communication between individuals and multidisciplinary teams [
The introduction of digital technologies offers one potential solution, allowing ready detection of the deteriorating patient, communication among health care workers, and immediate access to patient data in a user-friendly and appropriate format, thus improving outcomes. The embedding of digital technologies into health care is now a priority in the United Kingdom [
AKI—a sudden reduction in kidney function diagnosed by changes in serum creatinine [
We present a qualitative evaluation of the experiences of users and other health care professionals whose work was affected by the implementation of the new care pathway. We sought to characterize the impacts on staff of such automated alerting, mobile results viewing, and ready communication, with particular focus on their working practices and interprofessional relationships.
The digitally enabled pathway was designed and implemented at the Royal Free Hospital (RFH)—a large, acute, tertiary referral hospital providing a range of acute services (including a 34-bed intensive treatment unit and an inpatient nephrology service) in central London, United Kingdom. The care pathway has been described in detail elsewhere [
Before making the changes to the care pathway described here, pathology results were viewed by ordering clinicians in batch at the end of the working day using desktop computers. A message linking to local Web-based clinical guidelines was appended to any creatinine result suggestive of AKI in the electronic patient record (EPR), and such results were also communicated to the clinical area by biochemistry staff by telephone. In its early stages, AKI was typically managed independently by general acute care and various specialty teams; specialist input was requested through hospital pagers and telephone communication at the discretion of referring clinical teams.
Streams (DeepMind Technologies Ltd) is a mobile app deployed on iPhone operating system–enabled smartphones. It processes relevant routinely collected clinical and demographic data through secure integration with hospitals’ existing information systems; owing to the need for real-time event-driven data, Health Level Seven (version 2, Health Level Seven International) feeds were used for integration with the laboratory information management system and electronic medical record. It was first registered with the Medicines and Healthcare Products Regulatory Agency as a Class I, nonmeasuring, nonsterile medical device under the European Union Medical Device Directive (1993) on August 30, 2016. Future revisions of the device may be classified at a higher level under the new medical device regulation.
Streams analyzes serum creatinine results immediately and continuously, alerting clinicians in real time to all potential AKI cases as defined by the NHS England AKI algorithm. The app also provides clinicians with data relevant to AKI management, including a graphical trend view of serum creatinine, specific flags for the presence of life-threatening AKI complications (such as hyperkalemia), details of any previous AKI episodes, demographic information, and past medical history from coded Hospital Episode Statistics data. Videos demonstrating Streams functionality can be found on the DeepMind Health support website [
AKI alerts are sent in real time to a specialist clinical response team (henceforth, the
The AKI response team prioritizes patients for review according to the information available in Streams. Patients with life-threatening complications or deemed at high risk are reviewed immediately, whereas a case review within 2 hours is suggested for all other alerts. Upon review, a care protocol (based on existing best practice guidelines [
Although the nephrology team occasionally takes over patient care, overall responsibility for care rests with patients’ primary teams. AKI recovery is monitored remotely in-app. Realerting for AKI that has not recovered is enabled 48 hours after the first alert. However, worsening of AKI stage or the development of a new complication (eg, hyperkalemia) at any time results in a further notification. Follow-up reviews are undertaken by the AKI response team according to clinical judgement. A diagram outlining the pre- and postintervention care pathways is provided in
Pre- and postintervention care pathways. AKI: acute kidney injury.
Before deployment, Streams users attended training events and accessed a video users’ guide to both the Streams app and the clinical pathway. Feedback from the AKI response team was gathered during a 16-week pathway optimization period (January-May 2017), during which key adjustments to the Streams app and associated clinical pathway were made. The optimized care pathway has been deployed continuously at RFH since May 2017.
We undertook an exploratory case study approach to understand the implementation of Streams within a single site. Data collected included in-depth semistructured interviews and brief nonparticipant observations. The interview guide (included in the
Data were analyzed using a combination of inductive and deductive thematic analysis techniques [
The digitally enabled care pathway constituted a new standard service at RFH. Plans for the evaluation of the digitally enabled care pathway were independently reviewed by the University College London Joint Research Office. They directed that this study fell under the remit of service evaluation (rather than research), as per guidance from the NHS Health Research Authority [
Although interviews focused on the deployment of a pathway for patients with AKI, they also sought to more broadly characterize the impacts of automated alerting, mobile results viewing, and ready communication on staff, with a particular focus on their working practices and interprofessional relationships. In this respect, 3 central themes emerged.
This theme relates to respondents’ experiences of the benefits and drawbacks of real-time mobile access to patient data. The app provided automated alerts about patients’ kidney function and gave staff access to current and historical information, such as previous pathology and coded diagnoses. These functions were reported to save valuable time among participants from both teams:
Being able to look up the blood results for anyone in the hospital wherever you are is unparalleled. [...] it feels almost archaic these days, to go and see a patient and then go and sit down in front of a computer 15 minutes later. As a doctor, you have to integrate what you know about them at the time of seeing them. So if you could literally have this phone, look at the results, go and see them... Or even look at it while you are seeing them. [...] It must save at least - I don’t know if you could analyze it - but it must save at least a couple of hours in a day.
This in turn expedited rapid intervention for deteriorating patients, wherever they were in the hospital:
The speed at which it happened was impressive. [...] I happened to be in A&E [Emergency Department] and got the alert of someone with severe kidney injury. [...] The patient was admitted to [...] a specialist renal ward [...] within 2 or 3 hours, which I don’t think would have happened without the app. [...] I think it streamlines care and speeds up the time in which they get a specialist renal review.
I personally have noticed [...] patients who have flagged up on the app that the clinical management has been poor up to that point. When we get involved, or the renal team get involved, that management changes [...] It has definitely saved people’s lives.
Being able to access all the bloods for the patients in the hospital and to be able to be alerted to the sick ones and already know about them before we usually do... Sometimes you know about them before the crash bleep comes through. You turn up and you think, “That was actually the alert I was coming to see.”
Participants in both teams found alerts to be particularly valuable for patients whose lead consultant was not a physician:
The most value came from patients under [...] surgical patients, for whom the list of priorities for their clinicians are very different from what [physicians] look for when they are looking after a patient. For those [patients], getting a rapid alert about deranged renal function is very valuable.
The provision of results and real-time clinical alerts and team communication via mobile phones introduced workload for clinicians in a new modality. Overall, experienced clinicians were able to integrate AKI alerts into their existing duties, discriminating between high- and low-priority cases and using this information to adjust their current priorities:
I would intermittently [...] check it, like I would [...] check emails, [...] check it every hour or so, something like that. And within 5 minutes or so I could easily flick through the alerts and [...] identify which ones I needed to see. [...] I felt it was very easy to use, I think some people when they were trying to use it would try and respond immediately to every alert. I wouldn’t personally, I didn’t think that was the best way to do it. Intermittently checking it throughout the day, I managed to keep on top of things.
However, some more junior clinicians in both teams felt that the pathway created additional workload and suggested that clinical review might not be deliverable to all patients by the AKI response team as configured currently:
It does increase our work. Some days [...] we can have eight or nine referrals. But there is obviously a huge issue about workload for many people. But if we need to increase the size of our team because of this then that’s a good thing. And also it highlights [...] the acuity of our patients in our hospital. These patients are not straightforward.
Others pointed out that the added burden of the app was related to the volume of
...if the noise of the system could be reduced it would be a lot better. If [we were] able to get rid of all the nonsense alerts, that would be fantastic.
Some respondents from the nephrology team pointed out that although important information was now more readily available, this created additional anxiety because it was not clear who was primarily responsible for delivering timely care. Once again, the need to expand the responsive workforce was suggested as a way to mitigate the workload-associated stress:
...as Renal [Registrars], [...] you are always now, in the back of your head, thinking “I’ve got this other job to do.” And I think it does create... not anxiety that keeps you up at night... But it’s another anxiety when you already have enough anxiety! So I think even if it was available in the hands of more people, or we were a bit clearer that during times of people being unwell who are your own patients, you shouldn’t prioritise Streams people because they are under another team, then that’s fine. That’s one way of dealing with it.
In summary, although the digitally enabled care pathway was widely valued for increasing efficient access to patient information, thus facilitating better care, some respondents reported increased workload and anxiety.
The digitally enabled care pathway was designed to expedite identification of AKI so as to avoid deterioration and improve outcomes, meaning that staff who normally respond to critically unwell patients would now also attend patients at lower risk. There was a divergence of opinion among members of the AKI response team about the rationale for this approach. Respondents in both teams pointed out that the AKI algorithm identified deteriorating patients at an earlier stage than was possible through other means:
It’s a good thing from the point of view that I know there are patients that are potentially sick out there. [...] You could have an AKI and look relatively well initially. But [...] nobody would have known about those patients.
Others emphasized the benefits of early recognition for patient health and in terms of reducing the complexity of required interventions:
I think it does a good job. We pick up patients that would maybe sit for another day or so before we pick them up. It’s certainly beneficial. It is more work, but we might be saving ourselves work in a couple of days, we might have to do more stuff to catch up.
However, some respondents in the nephrology team did not see the point of being alerted to low-risk patients:
...you get a lot of AKI stage 1s. They build up. Looking through those and dismissing them each time is time consuming. The AKI stage 2 and 3 [alerts] are more helpful for me to look at, so I tend to just look at those and dismiss the stage 1s.
Some respondents in this team also pointed out that early identification could not necessarily be aligned to early intervention because of a lack of knowledge with respect to appropriate management:
The patients you definitely need to see are the patients that have acute renal failure with a creatinine of 300 or 400 [µmol/L] that and it’s going up - patients you’d normally want to see [...] The other patients [...] that have a rising creatinine, but the creatinine is not very high - it doesn’t mean that they don’t need to be seen necessarily. We are not trained as doctors to look after those sorts of patients.
Thus, the shift toward earlier detection highlights the need to consider the resources required to manage both early and late disease and the training needed to enable clinicians to effectively intervene at an earlier stage.
The final theme demonstrates how real-time data provision affected relationships between users of the digital intervention and the broader health system with which the users interacted, together with how these changes impacted upon beliefs, behaviors, and care delivery. These are described in 3 subcategories below.
First, the digital care pathway affected behaviors
I’ve found the [mobile phone] really useful because I’ve been able to message my team when I’m out seeing a patient, rather than finding a phone and to bleep them with and waiting for them to answer.
However, a disadvantage was also identified: junior members of the nephrology team do not currently use the Streams app. Unequal access to patient data occasionally reversed the usual direction of communication of information from junior doctor to consultant, which some consultants suggested impeded learning opportunities:
I think it’s important for [Junior Doctors] to understand what the decisions about their patients are. They have to be across the data. And that’s why I prefer getting information from them [...] We were in the position where I was telling the Juniors what the blood results were. It makes me uncomfortable and it makes them uncomfortable.
In addition, the visualization of each other’s triage decisions within the app (a feature specifically requested by users) revealed the hitherto unrecognized variations that exist in professional judgements. Respondents in both teams raised the fact that knowledge of others’ decisions sometimes confused rather than clarified the clinical decision-making processes of colleagues:
I was quite surprised about how other people triaged initially. I felt we’d be much more similar in our thinking, because when we talk about other things we do think similarly about other stuff. [...] I felt like - probably naively - that everyone would do what I did. And they didn’t at all.
Second, the digital pathway had an impact on relationships
So you might [...] call the team and say “we suggest you give some fluid,” but I don’t think it’s ethical to prescribe it yourself. After all, they might say, “Listen, he has heart failure.” So you can’t intrude.
However, there was an indication among several members of the PARRT team that this concern might be limited to communication between different specialty doctors:
I think the doctors have found it more difficult because in medicine there is this real model of, “[...] I don’t see this patient unless I’m asked to see them”. There’s this formality. [...] Nurses don’t think like that, people are used to us showing up. So it’s been easier for us to think of every patient that we see as our patient, our problem, our sick patients. I think that’s been easier for this team to absorb and deal with.
This is relevant because the type of information available also changed the professional group with whom this team directly liaised:
[I will] always [speak to] the nursing staff, because you are on their ward. It’s only polite and also you will generally be recommending frequencies of obs; they need to know what’s going on. And someone from the medical team. I would say the app is making me speak to more senior doctors more. [...] I’d be more likely to seek out a consultant and say “by the way, this person has alerted” and show them the app.
Third, the care pathway had an impact on the relationship between clinicians and their patients. In particular, several PARRT team members described that alerts identified patients at an earlier stage of AKI than was the case through established clinical pathways (eg, monitoring of vital signs). This may have led to an unexpected and evolving role for some members of the team. Several respondents described how the care pathway enabled them to help patients make informed decisions surrounding end-of-life care. For example:
Why do we have to talk about end of life just as I’m about to die? [...] We could plan. Every single person we’ve been referred today has a terminal disease. [...] Trying to move the decision making back, in a more timely way. [...] We are getting an alert before they have even triggered [via vital signs], so we can probably have a sensible conversation with a patient with capacity.
Qualitative and quantitative results from our mixed-methods service evaluation suggest that the digitally enabled care pathway has positive impacts on patient care [
This evaluation has a number of strengths. First, it benefits from the diversity of our respondent sample, presenting multiple perspectives on the intervention based on cultural differences between different professionals and teams. Second, the robust analysis uncovered issues that are likely to be generalizable to the implementation of other digital technologies in health care. Finally, our analysis team includes researchers from medical, public health, and psychology backgrounds, creating trustworthiness by encouraging debate and multidisciplinary interpretation.
Our evaluation has a number of limitations. First, we initiated interviews during the
Few studies of AKI alerting systems have been previously described. Alavijeh et al [
Our results are relevant to the design and evaluation of care pathways that involve automated alerts, mobile working, or the early deployment of specialist care. Such innovations will increasingly emerge with the application of machine learning [
Care Protocol, nursing advisory sticker, interview guide, and Royal Free Hospital committee members.
Matrix of quotes from interviews. Rows represent individual respondents, and columns represent categories which align to the basic principles of the intervention pathway.
acute kidney injury
electronic patient record
National Health Service
National Institute for Health Research
patient-at-risk and resuscitation team
Royal Free Hospital
RR and GB are in part supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust, and RR is an NIHR senior investigator. GR and HM are funded in part by the NIHR University College London Hospitals Biomedical Research Centre. OSA is in part supported by an NIHR academic clinical fellowship. The authors wish to thank the staff and patients of the RFH and the RFH data monitoring committee. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.
HM, CL, RR, CH, AK, TB, KA, DK, and MS initiated the project and the collaboration. CL conceived the care pathway. AC, CL, CM, JC, GJ, SS, and ME supported implementation. RR led the design of the evaluation with assistance from AC, CL, GR, HM, PM, and CN. AC and CL triaged alerts necessary for the evaluation. AC collected all necessary data that were analyzed by AC with assistance and oversight from GB and RR. AC, HM, RR, PM, CL, OSA, GB, and GR wrote the paper. All authors read and agreed to the final submission.
CL, HM, GR, and RR are paid clinical advisors to DeepMind. AC’s clinical research fellowship was part-funded by DeepMind. CL was a member of the National Institute for Health and Care Excellence clinical guideline 169 development group referenced in the paper. DeepMind remained independent from the collection and analysis of all data. HM coholds a patent on a fluid delivery device that might ultimately help in preventing some (dehydration-related) cases of AKI occurring. DeepMind was acquired by Google in 2014 and is now a part of the Alphabet group. The deployment of Streams at RFH was the subject of an investigation by the Information Commissioner’s Office in 2017. RFH has since published an audit completed to comply with undertakings following this investigation [