Design and implementation of a system for automated monitoring of adherence to evidenced-based clinical guideline recommendations

Background Clinical practice guidelines are systematically developed statements intended to optimize patient care. However, a gap-less implementation of guideline recommendations requires health care personnel not only to be aware of the recommendations and to support their content, but also to recognize every situation in which they are applicable. To not miss situations in which guideline recommendations should be applied, computerized clinical decision support could be given through a system that allows an automated monitoring of adherence to clinical guideline recommendation in individual patients. Objectives (1) To derive the requirements for a system that allows to monitor the adherence to evidence-based clinical guideline recommendations in individual patients, and based on these requirements, (2) to implement a software prototype that integrates clinical guideline recommendations with individual patient data and (3) to demonstrate the prototype's utility on a COVID-19 intensive care treatment recommendation. Methods We performed a work process analysis with experienced intensive care clinicians to develop a conceptual model of how to support guideline adherence monitoring in clinical routine and identified which steps in the model could be supported electronically. We then identified the core requirements of a software system for supporting recommendation adherence monitoring in a consensus-based requirements analysis within loosely structured focus group work of key stakeholders (clinicians, guideline developers, health data engineers, software developers). Based on these requirements, we implemented a prototype and demonstrated its functionality by integrating clinical data with a treatment recommendation. Results Based on our conceptual flow chart model of recommendation adherence monitoring in clinical routine, we identified four main requirements of a software system for automated support of recommendation adherence monitoring of in-hospital patients: (i) Ability to interpret guideline recommendations' semantics and logics, (ii) integration of clinical routine data from various underlying data structures, (iii) automatic adoption of new or updated guideline recommendations, and (iv) user interfaces optimized for distinct groups of users. Using a prototype implementation that fulfills these requirements, we demonstrate how such a system could be applied to monitor guideline recommendation adherence over time in clinical patients. Conclusions The four main requirements identified through our model-based analysis represent the most important aspects that need to be considered when developing a clinical decision support system for monitoring the adherence to evidence-based clinical guideline recommendations in individual patients. As each of the requirements corresponds to a different expertise (guideline development, health data engineering, software development, patient treatment), a modularized software architecture separated by area of required expertise seems favorable. Our prototype successfully demonstrates how such a modular architecture can be implemented to allow real-time monitoring of guideline recommendation adherence. This prototype, which we released as open source to invigorate collaboration, could serve as a basis for further development to integrate guideline recommendations with clinical information systems.


Abstract 28
Background 29 Clinical practice guidelines are systematically developed statements intended to optimize patient care. 30 However, a gap-less implementation of guideline recommendations requires health care personnel 31 not only to be aware of the recommendations and to support their content, but also to recognize every 32 situation in which they are applicable. To not miss situations in which guideline recommendations 33 should be applied, computerized clinical decision support could be given through a system that allows 34 an automated monitoring of adherence to clinical guideline recommendation in individual patients. 35 Objectives 36 (1) To derive the requirements for a system that allows to monitor the adherence to evidence-based 37 clinical guideline recommendations in individual patients, and based on these requirements, (2) to 38 implement a software prototype that integrates clinical guideline recommendations with individual 39 patient data and (3) to demonstrate the prototype's utility on a COVID-19 intensive care treatment 40 recommendation. 41

42
We performed a work process analysis with experienced intensive care clinicians to develop a 43 conceptual model of how to support guideline adherence monitoring in clinical routine and identified 44 which steps in the model could be supported electronically. We then identified the core requirements 45 of a software system for supporting recommendation adherence monitoring in a consensus-based 46 requirements analysis within loosely structured focus group work of key stakeholders (clinicians, 47 guideline developers, health data engineers, software developers). Based on these requirements, we 48 implemented a prototype and demonstrated its functionality by integrating clinical data with a 49 treatment recommendation. 50

51
Based on our conceptual flow chart model of recommendation adherence monitoring in clinical 52 routine, we identified four main requirements of a software system for automated support of 53 recommendation adherence monitoring of in-hospital patients: (i) Ability to interpret guideline 54 recommendations' semantics and logics, (ii) integration of clinical routine data from various underlying 55 data structures, (iii) automatic adoption of new or updated guideline recommendations, and (iv) user 56 interfaces optimized for distinct groups of users. Using a prototype implementation that fulfills these 57 requirements, we demonstrate how such a system could be applied to monitor guideline 58 recommendation adherence over time in clinical patients. 59

60
The four main requirements identified through our model-based analysis represent the most 61 important aspects that need to be considered when developing a clinical decision support system for 62 monitoring the adherence to evidence-based clinical guideline recommendations in individual 63 patients. As each of the requirements corresponds to a different expertise (guideline development, 64 health data engineering, software development, patient treatment), a modularized software 65 architecture separated by area of required expertise seems favorable. Our  to be associated with improved patient outcome, especially in the case of evidence-based 76 recommendations that were developed based on systematic reviews and appraisal of the available 77 evidence [6-8]. However, a gap-less implementation of clinical practice guideline recommendations 78 in daily routine work requires health care professionals not only to be aware of the existence of the 79 respective guideline recommendations, to understand and support their content, but also to correctly 80 recognize all situations in which specific recommendations should be applied [9]. 81 Meeting the latter requirement becomes particularly demanding in the interdisciplinary treatment of 82 patients with complex conditions that affect multiple organ systems, as it is often the case in critical 83 care medicine [10][11][12]. To ensure that all health care professionals have an active knowledge about all 84 guideline recommendations that apply in such situations and that they correctly recognize every 85 situation in which these recommendations should be applied can prove difficult. Thus, treatment in 86 critical care medicine is at a comparably high risk of deviating from guideline recommendations [10]. 87 Besides the multitude of simultaneously applicable guideline recommendations in critical care, 88 another aspect that can strongly affect guideline recommendation adherence is a high frequency of 89 changes in recommendations [11]. The coronavirus disease 2019 (COVID-19) pandemic presented an 90 exemplary situation in which dissemination and implementation of guidelines via conventional 91 processes struggled to hold pace with the rapid development of recommendations and the speed at 92 which recommendations were updated and changed over time [13,14]. Based on this representation, the aim of this study was (1) to collect and analyze the requirements for 104 providing clinical decision support via automated monitoring of individual evidence-based guideline 105 recommendation adherence, and (2) to design & implement a prototype that fulfills the requirements 106 and (3) to test the prototype's applicability on real patient data. 107 108 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. To derive the requirements for a software system to monitor the adherence to clinical guideline 111 recommendations, we first performed a work process analysis of the clinical processes that are to be 112 supported by the system. After identifying and structuring these processes, we determined which 113 subprocesses can be supported electronically and identified the requirements for how these 114 subprocesses can be supported by a software system. Based on these requirements we designed a 115 modular system architecture, which we implemented as an open-source prototype. To demonstrate 116 the utility of the prototype implementation, we applied it to monitor the adherence to a COVID-19 117 treatment guideline recommendation based on clinical data from a large European university hospital. To derive the requirements for the system, we first conducted a needs analysis for the users of the 123 system, the clinical staff. This needs analysis was conducted as a work process analysis, in which five 124 experienced clinicians contributed to flow chart modeling of how the adherence to guideline 125 recommendations would be monitored in clinical practice. Modeling was performed based on an 126 iterative feedback process, in which for each iteration the model was adapted until no further changes 127 were required. 128 Based on the work process model, we identified which parts of this process could be supported 129 electronically and developed a flow chart model of how clinicians would interact with an electronic 130 system to achieve the goal of monitoring patient-specific guideline recommendation applicability and 131 adherence. Identification of support potential and process modeling was again performed in an 132 iterative feedback process among the same group of clinicians together with a health software 133 architect. 134 Software design and prototype implementation CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 22, 2022.

165
Work process analysis 166 The needs analysis with the clinical staff resulted in a flow chart describing the work process of how 167 the adherence to guideline recommendations should be monitored (Figure 2 a). The core insight of the 168 work process analysis was that to evaluate and monitor guideline recommendation adherence, 169 clinicians always work at the ward level and examine each patient individually to see whether a 170 guideline recommendation applies and whether it is fulfilled. Identification of electronic support potential 175 Based on the work process model, the clinical staff together with a health software architect identified 176 which steps could be supported or covered by an electronic system. We identified nearly all steps as 177 susceptible to being taken over by a software system (Figure 2 a). The same group then determined 178 how this process should be supported electronically and developed a corresponding model of the 179 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 22, 2022. ; https://doi.org/10.1101/2022.07.18.22277750 doi: medRxiv preprint digitized work process (Figure 2 b). The most important core insight of this analysis was the necessity 180 to display the raw patient data that underlies the system's decision on recommendation 181 applicability/adherence, since clinicians using a software-based decision support system to monitor 182 recommendation adherence want to be able to examine the raw patient data that underlies the 183 system's decision on recommendation applicability/adherence, to make sure that this data and 184 decision is correct, as electronic health record data may contain errors. 185

Requirements analysis
186 For a computer system that should support the defined work process (Figure 2 a), we identified four 187 core requirements in a series of focus work group feedback rounds. 188 Requirement #1: The system needs to be able to decide whether a guideline recommendation 189 is applicable and whether a guideline recommendation is implemented for a specific patient The task of checking whether a guideline's recommendation is applicable and whether it is 191 implemented for an individual patient or not requires the system to be able to process both the 192 semantical (i.e., what is the meaning of the words used in the recommendation) and logical (i.e., which 193 of the words used in the recommendation define who the recommendation applies to and which 194 words define what is to be done or not to be done) content of the recommendation. Therefore, the 195 system needs to be provided with guideline recommendations in a format that is semantically correct, 196 complete, and unambiguous. 197 We here focus on evidence-based guideline recommendations, which makes the decoding of the 198 logical content of the recommendations particular easy: In the development process of evidence-199 based recommendations, it is standard practice to decompose the clinical question in consideration 200 according to the PICO framework (population/patients, intervention, comparison, outcomes,) [28]. 201 Therefore, in these recommendations, the patients to which the guideline recommendation is 202 applicable to (P in PICO) and the intervention (I in PICO) that is recommended are distinctly defined at 203 its best beginning with the systematic reviews supporting evidence-based guideline recommendations. 204 Regarding the decoding and interpretation of the semantical content of guideline recommendations, 205 we have developed a FHIR-based format for the representation of clinical practice guideline 206 recommendations to provide an interoperable, standards-based guideline recommendation exchange 207 format that fulfills the above requirements [23]. Moreover, a variety of formalisms for representing 208 guideline recommendations in a computer-interpretable way exist [29-31] and any of these could be 209 used, provided that they are able to represent guideline recommendations semantically correct, 210 complete and unambiguous. 211 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 22, 2022. and non-standardized data formats and data structures that differ between countries, hospitals or 215 even wards in the same hospital. Therefore, to make the system applicable across various existing IT 216 infrastructural settings, the second core requirement is that it needs to accept data in a standardized, 217 interoperable format, into which all proprietary data formats can be converted. Among data formats 218 that fulfill these requirements are the OMOP common data model (CDM) [32] or FHIR-based formats 219 (e.g. the US Core Profiles [33] or the German Corona Consensus dataset [34]). 220 Requirement #3: The system needs to automatically adopt changes in clinical guideline One aspect that especially complicates error-free manual implementation of guideline 228 recommendations in a software system is that such a task requires the expertise of at least two 229 different and highly specialized fields: the expertise of the medical subspecialty providing the guideline 230 recommendation and the software development expertise necessary for implementation into a 231 system. Having both at one's disposal for every single new or updated guideline recommendation is 232 difficult and costly. 233 Therefore, the system needs to adopt changes in guideline recommendations without requiring 234 changes in the system's software code itself. Instead, once changes in guideline recommendation are 235 released by the responsible medical societies or other appropriate sources, these changes should be 236 automatically adopted by the system without the requirement of any manual changes in the system's 237 software. 238 Requirement #4: The system needs to provide user interfaces optimized for distinct user groups 239 Different users of the clinical decision support system require different user interfaces, depending on 240 the specific work processes that are to be supported by the system. For example, medical or nursing 241 staff working on individual patients require a system that is highly integrated with their standard 242 patient data visualization used during the treatment process (e.g., the critical care information system 243 used on the ward) to allow to integrate the decision support seamlessly into the individual patient 244 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 22, 2022. ; care. In contrast, cross section staff like quality officers or supervising staff require more 245 comprehensive overviews over multiple patients simultaneously, with less integration with other 246 patient data, as their work processes that are to be supported by the system are more disconnected 247 from individual patient care. Therefore, the fourth core requirement is that the user interface of the 248 system must be customizable to meet the specific requirements that allow integration into the work 249 processes of the respective groups of users that are to be supported. 250

251
Considering the advantages of a modular system where each module corresponds to a specific 252 specialty, we decided for a software architecture with four main modules that correspond to the 253 previously described four main requirements that each require the involvement of only one of the 254 three stakeholder groups besides the software developers ( Figure 3; Table 1). 255 256 257  . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 22, 2022. ; https://doi.org/10.1101/2022.07.18.22277750 doi: medRxiv preprint The software was implemented as a containerized microservice architecture for effortless on-premises 262 deployment within individual hospitals' IT infrastructure, preventing any patient data from leaving the 263 hospitals' networks. We have designed the backend modules to be run within the same local network 264 with the only outside access possible to the user interface backend. Thus, in our prototype 265 implementation the user interface backend provides a user-level authentication to ensure only 266 authorized users can access the output of the system. Depending on the actual deployment scenario, 267 each of the microservices can be augmented by an authentication scheme. 268 For our exemplary prototype we have designed and implemented a user interface aimed to assist 269 supervising medical staff in their task to review whether specific guideline recommendations are 270 applicable and adhered to in individual patients that are currently treated in wards for which they are 271 responsible (Figure 4). The user interface designed for this specific task allows the user to select the 272 guideline recommendation to check and then, along with an overview over the patients currently 273 treated in the ward, gives a condensed evaluation to which of the patients this specific 274 recommendation is applicable and in which of the patients it is adhered to and in which it is not. 275 Furthermore, the user interface allows to view the patient data on which the guideline 276 recommendation evaluation was performed on to allow the clinician to review the evaluator's results. 277 The user interface is implemented as a dashboard website using RShiny  . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

306
In this paper, we demonstrated the system architecture and prototype implementation of a clinical 307 decision support system that automatically integrates clinical guideline recommendations with real-308 time clinical data to assist health care professionals by visualizing whether guideline recommendations 309 apply to individual patients and whether the guidelines recommendations have been followed in 310 individual patients or not. We described our stepwise approach for the development of the system, 311 including the core requirements which shaped our software architectural design, as well as our 312 prototype implementation and demonstrated the prototype's utility using a COVID-19 treatment 313 guideline recommendation on clinical data. 314 To evaluate our architecture and prototype, we implemented a recent clinical guideline 315 recommendation on treatment of patients with severe or critical COVID-19 disease and integrated the 316 recommendation with data from a large university hospital to analyze the guideline recommendation 317 adherence over time ( Figure 5). The integration of the machine-readable guideline recommendation 318 with clinical data could accurately detect the first and second wave of COVID-19 intensive care 319 treatments [40] and the successful implementation of the guideline recommendation for the second 320 wave, as seen by a >70% of relevant patients treated according to the recommendation. The non-100% 321 guideline recommendation implementation in our specific data set is primarily due to the fact that a 322 large part of COVID-19 patients treated in this university medicine center were transferred from other 323 hospitals and in parts have been treated with steroids in accordance with the guideline 324 recommendation before arriving on the intensive care unit of our hospital [41]. Such a situation in 325 which the recommended treatment has already taken place but was recorded in a different patient 326 data management system could be solved by increased interoperable data exchange between 327 different health care providers. 328 An automated integration of guideline recommendations with clinical data as done by our developed 329 systems holds several advantages: The system can provide a certain kind of decision support during 330 individual patient treatment by pointing to applicable guideline recommendations, which the treating 331 health care professionals might either not be aware of or which are known but whose applicability 332 might go unnoticed. Additionally, the monitoring of guideline recommendation adherence across 333 groups of patients provides possibilities for their use as quality and performance indicators [42,43] that 334 can easily be monitored in real-time using a system as we propose it here. Another similar aspect could 335 be the application of the system to monitor process implementation of new guidelines and to provide 336 clinical insights in applicability of guideline recommendations that are useful for guideline updates. 337 Independent from official clinical guidelines, the system can also easily be used to monitor hospital-338 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 22, 2022. ; https://doi.org/10.1101/2022.07.18.22277750 doi: medRxiv preprint specific treatment standards by formulating them as machine-readable guideline recommendations 339 and providing them via the guideline interface to the adherence evaluator. 340 We have designed our system to require machine-readable guideline recommendations, however, 341 guideline recommendations are currently available only in narrative, human-readable format. Thus, 342 individual guideline recommendations need to be converted in a machine-readable format first, adding 343 an extra amount of work. However, the specification of guideline recommendations in a machine-344 readable format enforces a precise and accurate formulation of guideline recommendation preventing 345 ambiguities, as these could not (easily) be resolved or understood by a software system. Additionally, 346 in contrast to converting human-readable recommendations into machine-readable 347 recommendations, the generation of precise human-readable formulations of a guideline 348 recommendation that is formulated in a machine-readable standard is a comparably simple task. 349 Therefore, specifying guideline recommendations from the start in a machine-readable format holds 350 multiple advantages and we therefore consider it a desirable change in the current practice of high 351 quality, evidence-based guideline recommendation development. An alternative approach to using a 352 machine-readable guideline recommendation specification could be the application of recent 353 advances of natural language processing (NLP) methods to allow the computer to "understand" and 354 process human-readable guideline recommendations [44,45]. However, any errors unknowingly 355 introduced by such an approach (e.g., due to imperfect "understanding" of the guideline 356 recommendation by the NLP algorithms) could have severe consequences on patient health and 357 outcomes. Therefore, we believe that the explicit statement of guideline recommendations in a 358 standardized machine-readable format is currently the more suitable choice. 359 In our prototype implementation we have used an EBMonFHIR-based representation of clinical 360 guideline recommendations that covers the whole development process of evidence-based guidelines, 361 from the underlying systematic review of available evidence over the rating of the individual evidence 362 to the final recommendation in a computer-interpretable way [23]. The computer-interpretable links 363 to the development process are particularly useful in meeting the requirement of the system to adapt 364 to updated guideline recommendations (requirement #3), as updates to recommendations based on 365 new evidence or re-appraisal of existing evidence can be automatically passed to the system without 366 the need of newly and manually converting human-readable guideline recommendations into a 367 representation formalism. While requirement #3 could in principle be met by any guideline 368 recommendation formalisms that is semantically correct, complete, and unambiguous, the CPG-on-369 EBMonFHIR representation offers an advantage especially with guideline recommendations that are 370 updated regularly, such as recommendations from living guidelines. Additionally, the CPG-on-371 EBMonFHIR representation could allow the users of the software system to evaluate the certainty of 372 the evidence and the evidence-to-decision process underlying individual recommendations. We did 373 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted July 22, 2022. ; https://doi.org/10.1101/2022.07.18.22277750 doi: medRxiv preprint not include this functionality in the prototype implementation, which, however, may become part of 374 a later extension stage of the system implementation. 375 In our approach, the user interface is separated from the rest of the software to allow user-and 376 context-specific implementation of user interfaces. We have provided a prototype as a dashboard 377 website to demonstrate its feasibility. However, in clinical practice during individual treatment it might 378 be desirable to integrate the suggestions of guideline recommendations into the clinical information 379 system (CIS) that is implemented on the ward. Due to the separation of user interface and backend in 380 our system, these integrations could be readily implemented depending on the CIS, e.g. as SMART-on-381 availability) and invite for cooperation and collaborative further development of the system [48]. 404 . CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.  CC-BY-NC-ND 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted July 22, 2022. ; https://doi.org/10.1101/2022.07.18.22277750 doi: medRxiv preprint