Clinical practice guideline recommendations are intended to optimize patient care by assisting the decision-making of health care professionals within specific clinical circumstances [1-3]. Thus, clinical practice guideline recommendations are among the most important potential clinical decision support (CDS) tools [4,5]. Considering and implementing such recommendations during patient management is expected to be associated with improved patient outcomes, especially in the case of evidence-based recommendations that were developed based on systematic reviews and appraisal of the available evidence [6-8]. However, a gapless implementation of clinical practice guideline recommendations in daily routine work requires health care professionals not only to be aware of the existence of the respective guideline recommendations, to understand and support their content, but also to correctly recognize all situations in which specific recommendations should be applied [9].

Meeting the latter requirement becomes particularly demanding in the interdisciplinary treatment of patients with complex conditions that affect multiple organ systems, as it is often the case in critical care medicine [10-12]. Ensuring that all health care professionals have active knowledge about all guideline recommendations that apply in such situations and that they correctly recognize every situation in which these recommendations should be applied can prove difficult. Thus, treatment in critical care medicine is at a comparably high risk of deviating from guideline recommendations [10].

Besides the multitude of simultaneously applicable guideline recommendations in critical care, another aspect that can strongly affect guideline recommendation adherence is the high frequency of changes in recommendations [11]. The COVID-19 pandemic presented an exemplary situation in which the dissemination and implementation of guidelines via conventional processes struggled to keep pace with the rapid development of recommendations and the speed at which recommendations were updated and changed over time [13,14].

To counter such difficulties and assist in the implementation of clinical guideline recommendations using computerized CDS, various machine-readable guideline recommendation formalisms have been developed [15-22]. However, these formalisms focus on representing finalized guideline recommendations and do not consider the systematic development process from which evidence-based guideline recommendations are derived. We have recently developed a Fast Healthcare Interoperability Resources (FHIR)–based formalism for the computer-interpretable representation of the whole guideline recommendation development process by developing systematic reviews of primary studies, rating the certainty of the available body of evidence, and finally applying evidence to decision frameworks to derive the final recommendation, called Clinical Practice Guidelines on Evidence-Based Medicine on FHIR (CPG-on-EBMonFHIR; [23]). This formalism is geared toward separate evidence-based guideline recommendations, which are often simple rules linking a specific condition to a specific action, as they are usually based on studies that focus on specific interventions and limited options rather than complex clinical pathways because of the logical complexity that can be studied in randomized trials.

We aimed to (1) collect and analyze the requirements for providing CDS via automated monitoring of individual evidence-based guideline recommendation adherence, (2) design and implement a prototype that fulfills the requirements, and (3) test the prototype’s applicability on real patient data.

To derive the requirements for a software system to monitor adherence to clinical guideline recommendations, we first performed a work process analysis of the clinical processes that are to be supported by the system (Figure 1). After identifying and structuring these processes, we determined which subprocesses can be supported electronically and identified the requirements for how these subprocesses can be supported by a software system. On the basis of these requirements, we designed a modular system architecture that we implemented as an open-source prototype. To demonstrate the utility of the prototype implementation, we applied it to monitor adherence to a COVID-19 treatment guideline recommendation based on clinical data from a large European university hospital.

To derive the requirements for the system, we first conducted a needs analysis for the users of the system, the clinical staff. This needs analysis was conducted as a work process analysis in which 5 experienced intensive care clinicians contributed to the flowchart modeling of how adherence to guideline recommendations would be monitored in clinical practice. The modeling was performed based on an iterative feedback process as follows:

During the development of the conceptual model, there were some disagreements among the 5 clinicians involved in the process. These disagreements arose when discussing the steps and level of detail that should be included in the model based on their experience and individual clinical practice. However, these disagreements were reconciled through open and constructive dialog among the clinicians. The goal was to reach a consensus to ensure that the conceptual work process model accurately reflected general clinical practice.

On the basis of the work process model, we identified which parts of this process could be supported electronically and developed a flowchart model of how clinicians would interact with an electronic system to achieve the goal of monitoring patient-specific guideline recommendation applicability and adherence. The identification of support potential and process modeling was again performed in an iterative feedback process among the same group of clinicians, together with a health software architect.

To identify the requirements of the software system for monitoring individual recommendation applicability and adherence, we performed a comprehensive needs analysis for the system by involving key stakeholders:

Stakeholders were recruited by convenience from participants at the senior level of their respective field within the COVID-19 Evidence Ecosystem project of the Federal Network of University Medical Centers in Germany [24]. They were approached individually and requested to participate. No compensation was offered to participants. We required at least 2 participants per stakeholder group.

The software prototype was implemented using an agile, rapid application development approach. The architecture followed a microservice pattern to allow efficient separation of concerns and scalable and exchangeable deployments of the system within heterogeneous clinical IT infrastructures. Each container exposes a RESTful application programming interface (API) specified according to the OpenAPI 2.0 standard [25]. Backend modules were implemented in Python 3.8 (Python Software Foundation) and the frontend modules using RShiny (RStudio) [26].

To demonstrate the utility of the prototype, we connected it to anonymized clinical data from a large university hospital (Charité Universitätsmedizin Berlin, Germany) and integrated a recent strong evidence-based guideline recommendation for the treatment of patients with severe or critical COVID-19 [27].

The use of anonymized clinical data for research was approved by the local ethics committee (Ethikausschuss 4 am Campus Benjamin Franklin, Charité Universitätsmedizin Berlin, Chairperson Prof R Stahlmann, Application Number EA4/008/19, approval date: February 6, 2019, amendment date: May 14, 2020).

To ensure that our system effectively supports the work processes of intensive care physicians, we conducted a comprehensive needs analysis with a group of key stakeholders. This included clinical staff made up of experienced intensive care physicians, as they are the primary users of the system, clinical practice guideline developers, health data engineers familiar with hospital IT infrastructure, and software developers. Together, they created a flowchart outlining the process of monitoring adherence to guideline recommendations (Figure 2A). The core insight of the work process analysis was that to evaluate and monitor guideline recommendation adherence, clinicians always work at the ward level and examine each patient individually to see whether a guideline recommendation applies and whether it is fulfilled.

On the basis of the work process model, the clinical staff, together with a health software architect, identified which steps could be supported or covered by an electronic system. We identified nearly all steps as susceptible to being taken over by a software system (Figure 2A). The same group then determined how this process should be supported electronically and developed a corresponding model of the digitized work process (Figure 2B). The most important insight of this analysis was the necessity to display the raw patient data that underlies the system’s decision on recommendation applicability and adherence because clinicians using a software-based decision support system to monitor recommendation adherence want to be able to examine the raw patient data that underly the system’s decision on recommendation applicability and adherence to ensure that these data and decisions are correct, as EHR data may contain errors.

For a computer system that should support the defined work process (Figure 2A), we identified 4 core requirements in a series of focus work group feedback rounds.

To demonstrate the utility of our prototype implementation, we have specified a recent evidence-based recommendation for the administration of dexamethasone to critically ill patients with COVID-19 from the guideline for inpatient treatment of patients with COVID-19 as a machine-readable guideline recommendation [44-46]. We connected the prototype to the critical care information systems and clinical information systems (CISs) of Charité Universitätsmedizin Berlin using a site-specific implementation of the clinical data interface. An exemplary time-dependent analysis of the applicability and adherence to guideline recommendations is shown in Figure 5 [47,48].

In this study, we demonstrated the system architecture and prototype implementation of a CDS system that automatically integrates clinical guideline recommendations with real-time clinical data to assist health care professionals by visualizing whether guideline recommendations apply to individual patients and whether the guidelines recommendations have been followed in individual patients. We described our stepwise approach for the development of the system, including the core requirements that shaped our software architectural design as well as our prototype implementation and demonstrated the prototype’s utility using a COVID-19 treatment guideline recommendation on clinical data.

To evaluate our architecture and prototype, we implemented a recent clinical guideline recommendation on the treatment of patients with severe or critical COVID-19 and integrated the recommendation with data from a large university hospital to analyze guideline recommendation adherence over time (Figure 5 [47,48]). The integration of the machine-readable guideline recommendation with clinical data could accurately detect the first and second waves of COVID-19 intensive care treatments [49] and the successful implementation of the guideline recommendation for the second wave, as seen by >70% of relevant patients treated according to the recommendation. The non-100% guideline recommendation implementation in our specific data set is primarily because a large proportion of patients with COVID-19 treated in this university medicine center were transferred from other hospitals and were treated with steroids in accordance with the guideline recommendations before arriving at the intensive care unit of our hospital [50]. Such a situation in which the recommended treatment has already taken place but was recorded in a different patient data management system could be solved by increased interoperable data exchange between different health care providers.

An automated integration of guideline recommendations with clinical data, as done by our developed systems, has several advantages: the system can provide a certain kind of decision support during individual patient treatment by pointing to applicable guideline recommendations, which the treating health care professionals might either not be aware of or which are known but whose applicability might go unnoticed. In addition, the monitoring of guideline recommendation adherence across groups of patients provides possibilities for their use as quality and performance indicators [51,52] that can easily be monitored in real time using a system, as we propose here. Another similar aspect could be the application of the system to monitor the process implementation of new guidelines and to provide clinical insights into the applicability of guideline recommendations that are useful for guideline updates. Independent of official clinical guidelines, the system can also easily be used to monitor hospital-specific treatment standards by formulating them as machine-readable guideline recommendations and providing them via the guideline interface to the adherence evaluator.

One of the key considerations in the design of our system is the separation of the contents of the clinical guidelines from the CDS software that integrates them with patient data. This approach has several advantages, including the separation of concerns; guideline developers can focus on the development of guidelines, whereas software developers can focus on the development of software. This allows for rapid updates of the guidelines, particularly for living guidelines that are continuously updated. However, this approach also has drawbacks. For example, rapidly changing guidelines may cause confusion for clinical staff if the system reports that patients are not being treated according to the current guideline simply because a new version of the guideline has been released and the treatment recommendation has changed. In addition, if the guidelines contain new decisions, items, or conditions that have not been covered by the execution engine code, the execution engine will not be able to process these updated guidelines.

We designed our system to require machine-readable guideline recommendations; however, guideline recommendations are available only in narrative, human-readable formats. Thus, individual guideline recommendations must first be converted into a machine-readable format which adds an extra amount of work. However, the specification of guideline recommendations in a machine-readable format enforces a precise and accurate formulation of guideline recommendations, which prevents ambiguities, as these cannot (easily) be resolved or understood by a software system. In addition, in contrast to converting human-readable recommendations into machine-readable recommendations, the generation of precise human-readable formulations of a guideline recommendation formulated in a machine-readable standard is a comparably simple task. Therefore, specifying guideline recommendations from the start in a machine-readable format has multiple advantages, and we consider it a desirable change in the current practice of high-quality evidence-based guideline recommendation development. An alternative approach to using a machine-readable guideline recommendation specification could be the application of recent advances in natural language processing methods to allow the computer to understand and process human-readable guideline recommendations [53,54]. However, any errors unknowingly introduced by such an approach (eg, because of imperfect understanding of the guidelines by natural language processing algorithms) could have severe consequences for patient health and outcomes. Therefore, we believe that the manual and explicit statement of guideline recommendations in a standardized machine-readable format, as used by our system, is currently the most suitable choice.

In our prototype implementation, we used an EBMonFHIR-based and CPG-on-FHIR-based representation of clinical guideline recommendations that covers the entire development process of evidence-based guidelines, from the underlying systematic review of available evidence on the rating of the individual evidence to the final recommendation in a computer-interpretable manner [23]. Computer-interpretable links to the development process are particularly useful in meeting the requirements of the system to adapt to updated guideline recommendations (requirement 3), as updates to recommendations based on new evidence or reappraisal of existing evidence can be automatically passed to the system without the need for new and manual conversion of human-readable guideline recommendations into representation formalism. Although requirement 3 could, in principle, be met by any guideline recommendation formalism that is semantically correct, complete, and unambiguous, the CPG-on-EBMonFHIR representation offers an advantage, especially with guideline recommendations that are updated regularly, such as recommendations from living guidelines. In addition, the CPG-on-EBMonFHIR representation could allow users of the software system to evaluate the certainty of the evidence and the evidence to decision process underlying individual recommendations. We did not include this functionality in the prototype implementation; however, it may become part of a later extension stage of the system implementation. Although a key idea behind the system is that guidelines can be updated efficiently and independently of human intervention, we have not performed a formal analysis of compute time dependencies for updates to the guidelines, as our focus was on developing the system requirements and demonstrating the technical feasibility of the system. However, if the guidelines are formulated in the FHIR-based standardized format, they can be updated, in principle, with time lags arising only from conceptually new recommendations or human processes and not from technical limitations.

This project aimed to create a system that specifically supports intensive care physicians’ work processes. However, we understand that different specialties or user groups may have unique approaches and techniques for managing various conditions and therefore may require different interfaces and work process integrations. To address this issue, the proposed system was designed with a modular architecture. This allows for the development of diverse user interfaces for various user groups, specialties, and disease models while maintaining the core functionality of checking guideline recommendations against patient data. This approach eliminates the need to build a new system for each application. Here, we developed a single prototypical user interface for demonstration purposes in the form of a dashboard website to demonstrate its feasibility. However, in clinical practice, it might be desirable to integrate the suggestions of guideline recommendations into the CIS implemented in the ward. For example, the system could be integrated into an EHR system user interface by highlighting laboratory values that fall outside of ranges supported by evidence-based recommendations and then providing the actual narrative recommendation and its underlying evidence on demand to the user. Owing to the separation of the user interface and backend in our system, these integrations can be readily implemented depending on the CIS, for example, as SMART-on-FHIR CDS hooks [55,56], if the CIS provides a SMART-on-FHIR interface.

In terms of the range of applicability of our system, we believe that it has the potential to be used in various medical specialties and clinical settings. The system is based on the use of evidence-based guideline recommendations, which are applicable across many different areas of medicine. Specifically, the system was designed to support the implementation of evidence-based guideline recommendations in critical care medicine, where the complexity and multiplicity of simultaneously applicable recommendations can make it challenging for health care professionals to correctly recognize every situation in which these recommendations should be applied. However, the system architecture is modular and independent of a particular user interface, which means that it can be adapted to support the implementation of guideline recommendations in other specialties and settings, such as primary care. Therefore, the range of applicability of the software can be broadened to other specialties and care settings. The main focus of the system is guideline-based CDS, and the actual implementation and integration of the system into specific clinical workflows are left to concrete implementation projects. In general, any guideline recommendation that can be checked using electronically recorded data can be used by a system such as the one demonstrated here.

We designed our software system to retrieve guideline recommendations from a centralized repository that could be hosted by medical societies or national standardization organizations. This allows societies or organizations to develop guideline recommendations independently of our system, thereby focusing on their expertise. Once new or updated guideline recommendations are published by medical societies or organizations on their servers, they can be automatically retrieved using our system and integrated with clinical data. However, a checkpoint in this process should be established, where health care professionals of individual hospitals first review new or updated guideline recommendations retrieved from the central repository before releasing them for implementation in their hospital. This helps mitigate risks that arise if the central guideline recommendation server is compromised by malicious attackers and ensures that the guideline recommendations implemented in the hospitals are in accordance with the hospitals’ policies.

The primary goal of the study was to derive the requirements for a system for automated guideline-based CDS and to develop a prototype and evaluate its technical feasibility. Although we have not yet conducted a formal evaluation of the system’s impact on clinical practice and patient outcomes, we believe that the system has the potential to improve patient care by assisting health care professionals in making informed decisions based on the latest evidence-based guideline recommendations. Studies have consistently shown that the implementation of guideline-based CDS systems can potentially improve the quality of care [57], increase guideline adherence [58,59], and positively impact the care process [60,61]. However, it should be noted that larger trials are needed to confirm these findings and that it is crucial to improve the interoperability of CISs to establish a widespread use of CDS systems, such as those developed by us [59]. The system developed in this study could help overcome some of the challenges commonly encountered in the implementation of guideline-based CDS systems. Specifically, the system’s ability to automatically process updated machine-readable guidelines as they are published in appropriate repositories and integrate them with standardized patient data could improve the timeliness and completeness of decision support, which, in turn, could lead to more accurate and consistent patient care and potentially better patient outcomes. However, further research is needed to confirm this potential impact and assess the system’s effectiveness of the system in different clinical settings.

Developing a system similar to ours, which integrates guideline recommendations with real-time clinical data, is a complex task that requires expertise from a variety of different fields. In addition to guideline experts, it also requires input from clinical experts, as well as specialized expertise in areas such as implementation, clinical data management, and IT infrastructure. The COVID-19 pandemic provided a unique opportunity for us to develop such a system and enabled us to bring together a wide range of expertise within the framework of a well-funded federal project, including the EBM-on-FHIR initiative, the German Association of the Scientific Medical Societies, which maintains the central German guideline register, and Cochrane Germany, to successfully develop and implement our system, thanks to the collaborative efforts of all stakeholders involved. Other initiatives aimed at improving the implementation of evidence-based guidelines in clinical practice include the CDS Connect project, led by the Agency for Healthcare Research and Quality of the US Department of Health [62], which focuses on developing and promoting the use of standards-based, computable clinical guidelines and knowledge artifacts in CDS systems. Another initiative is the Guideline Definition Language of the openEHR foundation, which allows expressing the decision logic for integration with the openEHR reference model [63]. These initiatives have similar goals to our system but focus on the development and sharing of computable knowledge artifacts, whereas our system focuses on the automated integration of evidence-based guidelines with patient data. Another initiative, on which we have built on, is the CPG-on-FHIR project, which provides an extensive methodology on how to represent guidelines in a computer-interpretable fashion [64], leveraging the Clinical Quality Language for creating reusable computable knowledge artifacts; however, to our knowledge, no system has been developed to integrate these guidelines with patient data. However, an execution engine for Clinical Quality Language artifacts is currently being developed [65].

The study and CDS system developed here have certain limitations that should be considered when evaluating its usefulness and applicability in clinical practice.

First, the system is currently focused on evidence-based guideline recommendations, which are simple rules linking specific conditions to specific actions, as they are typically derived from clinical studies with specific interventions and a limited number of options. Complex multistep clinical paths, consisting of decision points and possible actions that require user feedback, are not currently supported by the system. However, the CPG-on-FHIR standard on which we built the CPG-on-EBMonFHIR representation can represent complex multistep guidelines, and our system may be developed to support these more complex clinical workflows in the future [23,64].

Second, the system was designed to work with evidence-based guideline recommendations that are represented in a machine-readable format. However, evidence-based guideline recommendations have been developed for human-readable narrative statements. Representing them in a machine-readable format, such as EBMonFHIR or CPG-on-FHIR, poses a series of challenges and requires a significant amount of manual work and expertise from several groups of experts, such as guideline developers, knowledge engineers, and clinical experts [66]. In addition, not all guideline recommendations are suitable for representation in this format, which may limit the applicability of the system in certain situations. However, we believe that with ongoing international efforts to standardize health care data and interoperability with projects such as EBMonFHIR and the increased usability of artificial intelligence–based support tools for knowledge acquisition and representation, computable evidence will be available at scale in the future, which will minimize the need for manual translation of narrative guidelines into computer-interpretable representations.

Third, the system can only handle recommendations that can be evaluated using data from the hospital’s EHR system, which may limit its applicability in situations where other types of data or information are needed to make clinical decisions (eg, clinician instrumentation, patient preferences, or infrastructural settings).

Fourth, the system cannot consider uncertainties in the recommendations, which may impact its ability to provide appropriate guidance in certain cases. Furthermore, if the data used by the system are incorrect, this may lead to false alarms, which could negatively affect the trust of clinical staff in the system.

Fifth, the system currently cannot ask for feedback or input from human users. It is fully automated, which may lead to a lack of flexibility in certain cases.

In conclusion, although the CDS system described in this paper has the potential to improve adherence to guideline recommendations in clinical practice, the limitations described earlier should be considered when evaluating its usefulness and applicability in real-world settings.

In this study, we designed a system that integrates guideline recommendations with real-time clinical data to evaluate adherence to individual guidelines and to develop a functional prototype. The proposed system has advantages for both the individual treatment of patients, as clinical guidelines condense the current state of medical knowledge into treatment recommendations and for the quality management and monitoring of patient treatment standards in hospitals, which has the potential to improve the implementation of evidence-based guidelines and ultimately improve patient outcomes. However, further studies are required to quantify the specific impact of the system on patient outcomes. In addition, further research is needed to evaluate the resource effectiveness of the system and its potential for implementation in different clinical settings and specialties. We have specified a modular software architecture where each module corresponds to a particular area of expertise, allowing experts from different fields (guideline developers, software engineers, medical data engineers, and health care professionals) to work independently and focus on their area of expertise. We have released the source code of our system under an open-source license (see Data Availability) and invited for cooperation and collaborative further development of the system [67].