Improving the Development and Implementation of Audit and Feedback Systems to Support Health Care Workers in Limiting Antimicrobial Resistance in the Hospital: Scoping Review

Background: For eHealth technologies in general and audit and feedback (AF) systems specifically, integrating interdisciplinary theoretical underpinnings is essential, as it increases the likelihood of achieving desired outcomes by ensuring a fit among eHealth technology, stakeholders, and their context. In addition, reporting on the development and implementation process of AF systems, including substantiations of choices, enables the identification of best practices and accumulation of knowledge across studies but is often not elaborated on in publications. Objective: This scoping review aims to provide insights into the development and implementation strategies for AF systems for a real-world problem that threatens modern health care—antimicrobial resistance—and provide an interdisciplinary conceptual framework that can serve as a checklist and guidance for making informed choices in the development and implementation of future AF systems. Methods: A scoping review was conducted by querying PubMed, Scopus, Web of Science, IEEE Xplore Digital Library, and Embase ( ≥ 2010) for studies describing either the development or implementation process, or both, of an AF system for antimicrobial resistance or infections in hospitals. Studies reporting only on effectiveness or impact were to capture relevant AF systems and development and implementation constructs. Thereby, it provides a concrete checklist and overall guidance that supports the professionalization and harmonization of AF system development and implementation. For the development and implementation of future AF systems and other eHealth technologies, researchers and health care workers should be supported in selecting and integrating TMFs into their development and implementation process and encouraged to explicitly report on theoretical underpinnings and the substantiation of choices.


Introduction Background
Audit and feedback (AF) is a common reflective approach for various health care targets; however, the reported effects are small to moderate [1]. With the increase in electronically available data in health care, there is great potential for electronic AF systems [2]. The effectiveness of AF systems depends on the targeted behavior and the content, delivery, and context of AF and the system [1, [3][4][5][6]. These constructs are often studied after AF system development and implementation to evaluate strategies and their ingredients for success [7][8][9][10]. In the literature, less attention has been paid to the development and implementation processes of AF systems [3], as is also common in the broader field of eHealth [11,12]. The development process of eHealth can refer to the entire iterative process of developing an eHealth technology, from predesign and design to implementation and (summative) evaluation [13]. However, in this study, we focus on the process from predesign and design (referred to hereafter as development) to implementation, including formative evaluation cycles. This allows us to focus on the early stages of implementation and development that are truly intertwined, as potential implementation issues (eg, limited eHealth skills) should be accounted for early in the development process to avoid well-known pitfalls of stakeholder and context disregard [14]. These phases are entwined by iterative formative evaluation cycles that provide ongoing information on how to improve both the eHealth technology and the development process taking [13].
Development and implementation are essential to gain a profound understanding of relevant stakeholders, their thinking and work processes, and their context (including implementation factors). Without this understanding, a misfit among technology, context, and people is likely to occur, which decreases the effectivity and efficiency of eHealth in practice [13]. It is crucial to consider these constructs from the start of the development and implementation process to avoid common pitfalls in current AF, such as top-down expert-driven audits with feedback at the hospital level rather than personalized, actionable feedback that supports health care workers (HCWs) in improving the quality and safety of health care [15,16].
The application of theories, models, and frameworks (TMFs) is advocated as an integral part of eHealth development and implementation as it identifies what works for whom, why, how, and when, likely resulting in eHealth technology that achieves the desired outcomes [17]. Colquhoun et al [18] and Tuti et al [2] reported that only 9% (n=140) and 29% (n=7) of the included studies in their systematic reviews explicitly used theory to inform AF development and design. Therefore, implicit assumptions about AF working mechanisms and effective AF systems have driven AF development. Although these assumptions might hold true, they were not informed by theory [18,19], whereas there is a clear link between TMFs and eHealth intervention effectiveness [20,21].
To study the development and implementation of AF, this scoping review focuses on a real-world, wicked problem-antimicrobial resistance (AMR). AMR poses an increasing threat to human health and the durability of modern health care [22]. By 2050, AMR is expected to cause more yearly deaths worldwide than cancer currently does [23]. Antimicrobial and diagnostic stewardship programs and infection control programs form an integrated approach of AMR prevention measures (APMs) that aim to reduce and prevent the burden of AMR in hospitals [24]. Previous studies on HCWs' needs for APM support showed that changing HCWs' beliefs about their contribution to limiting AMR should be an important aim of APM strategies rather than merely focusing on raising AMR awareness or influencing ad hoc decisions [25,26]. To do so, learning through reflective cycles provides the opportunity to change HCWs' behaviors by giving them insight into their own behavior and improvement possibilities [15,27]. Therefore, AF for APM (APM-AF) is a promising strategy in the fight against AMR, although it is currently underused and understudied in the field of AMR [7].

Objectives
There seems to be no standardized way of (theoretically) substantiating choices for and reporting on the development and implementation of AF systems, which hinders the identification of best practices and knowledge accumulation [10,43]. Whereas other reviews on AF have mainly focused on the effectiveness of AF systems [1,2], this scoping review focuses on the development and implementation process to further harmonize and professionalize AF system development and implementation. The aim of this study is to gain insight into the development and implementation strategies for APM-AF systems by answering the following research questions: 1. What studies have been conducted so far to study the development and implementation of APM-AF systems? 2. What TMFs are used and described in studies on the development and implementation processes of APM-AF systems? 3. What information has been reported on APM-AF systems, and how are choices substantiated? 4. What information has been reported on the development and implementation processes of APM-AF systems, and how are choices substantiated? 5. What are the lessons learned for the development and implementation of APM-AF systems?
To allow for an evidence synthesis of information on the development and implementation of APM-AF, and because of the explorative aim and research questions in this study, a scoping review is preferred over a systematic literature review [44,45]. This scoping review provides an interdisciplinary conceptual framework that supports researchers, HCWs, and policy makers to make informed choices in APM-AF system development and implementation, with the aim of reducing the burden of AMR and improving the quality and safety of health care.

Methods
The PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist was used to report on this scoping review without a prior registered review protocol [46]. This scoping review was designed by a multidisciplinary research team comprising AMR and eHealth experts.

Eligibility Criteria
Studies were included if (1) they described the development and implementation process of an AF system (including monitoring and surveillance systems), (2) the system provided feedback to HCWs, and (3) the system targeted AMR and infections in hospitals. A more elaborate description of development and implementation is provided in Multimedia Appendix 1 [13]. We define AF systems as any system that comprises AF, wherein at least one of them (audit or feedback) is delivered or enhanced through the internet and related technologies [47]. As reporting on eHealth development and implementation processes is highly heterogeneous, there were no requirements for specific TMFs, methods, or eHealth technologies. Reviews and poster abstracts were excluded, as were studies outside the hospital setting. Evaluation studies that only reported on APM-AF effectiveness and impact without reporting on development and implementation were excluded. However, constructs of formative evaluation (defined as "activities throughout the entire development process that provide ongoing information on how to improve the development process, outcomes of activities and eHealth technology" [13]) were included, as it is intertwined throughout the eHealth development and implementation process. A full list of eligibility criteria can be found in Multimedia Appendix 1.

Data Charting Process
To chart the data, JK created a data extraction form (Multimedia Appendix 2 [2][3][4][5]18,43,[48][49][50]) in Microsoft Excel. The general study characteristics extracted were first author, year, journal, country, study aims, targets and target groups, study design and methods, and theoretical underpinning. Given the heterogeneous study approaches and theoretical underpinnings of the included studies, a comprehensive overarching conceptual framework was needed to systematically analyze relevant constructs. The conceptual framework was grounded in often-used TMFs and best practices from various scientific perspectives on AF [3][4][5]18] and for the description, development, and implementation of eHealth interventions in general [2,43,[48][49][50]. These TMFs and best practices were merged via an iterative axial and selective coding process by JK and NBJ. Thereby, overlapping and complementary constructs from various scientific perspectives were revealed. To structure all constructs, a distinction was made between constructs for APM-AF systems (n=41; research question 3) and constructs for development and implementation (n=35; research question 4).
The data extraction form was discussed within the research team, piloted, and iteratively refined throughout the assessment process. Note that this conceptual framework should be merely seen as an analytic framework for an organized way of thinking about and reporting on APM-AF systems from various

Synthesis of Results
The main researcher (JK) read all full texts and systematically extracted the data using the data extraction form. Studies were scored with a + for a comprehensive, ~for an incomplete, and -for a missing description for each data field. Descriptions were copied from the studies and further summarized per data field by describing variations among studies (ie, axial coding). In this process, data fields described by none of the studies were omitted (Multimedia Appendix 2), and other overlapping fields were combined. This reduced the number of data fields for APM-AF systems to 29. The translation and summarization of the extracted data into results were discussed in various rounds within the research team. Owing to the heterogeneity and qualitative nature of the included study designs, the richness and relevance of the (contextual) information were believed to be more important than study quality. Therefore, no quality appraisal was performed [51].

Study Selection
The literature search resulted in 3592 potentially relevant abstracts. Of the 3592 papers, after removing 1467 (40.84%) duplicates, 2125 (59.16%) unique titles and abstracts were assessed (Figure 1), which resulted in the eligibility assessment of 58 (1.61%) full texts. The main reasons for exclusion were a lack of information on development or implementation and evaluation studies (without reporting on development or implementation).

Study Characteristics
In total, of the 58 papers, 12 (21%) were included in this review (2012-2020), mostly from PubMed, Scopus, and Web of Science. Publications came from Northern American (6/12, 50%) or European (4/12, 33%) countries and Australia (2/12, 17%). Included studies stemmed from journals in various research fields (eg, infections or implementation science). Studies described APM-AF systems that were either in (preparation of) development or already implemented in practice, resulting in a wide variety of study aims, study targets, target groups, study designs, and used methods (Table 1).

Study Aims
Of the 12 studies, 4 (33%) primarily focused on development, 4 (33%) on implementation, and 4 (33%) described both development and implementation. However, development and implementation appeared to be undefined concepts, with implementation studies describing the development and design constructs and development studies paying attention to implementation constructs. Studies aimed at describing APM-AF system development focusing on (1) the integration of TMFs (eg, Feedback Intervention Theory), (2) AF content and presentation (eg, feedback gamification), or (3)

Study Targets and Target Groups
Of the 12 studies, 11 (92%) focused on one of the APM (ie, diagnostic stewardship programs, antimicrobial stewardship programs, or infection control programs), whereas 1 (8%) targeted multiple APM. The target groups comprised a variety of HCWs (both frontline staff and AMR experts; 8/12, 67%) and in some studies, administrators and managers (4/12, 33%) as well.

Study Design and Methods
Most studies (10/12, 83%) used multiple methods, complementing quantitative (eg, questionnaires) with qualitative data (eg, observations, interviews, and focus groups). Approximately 17% (2/12) of the studies were fully qualitative, relying on interviews and focus groups.

Theoretical Underpinning Described by Studies
Most studies (9/12, 75%) described the theoretical underpinnings of their APM-AF system or study approach (Table 1) [52]. Substantiations of choices on APM-AF systems were scarce; few studies substantiated their choices, which were either theory informed (eg, providing group-level feedback) or based on findings from the studies themselves (ie, revisions based on formative evaluation).

Conceptual Framework for APM-AF Development and Implementation
The conceptual framework, which is based on often-used TMFs and best practices for AF and eHealth interventions, is shown in Table 2 (APM-AF system constructs) and Table 3 (development and implementation constructs) and in Multimedia Appendix 2 in more detail.      The construction of the comprehensive overarching conceptual framework revealed the added value of including multiple perspectives, as 48% of constructs were complementary (ie, covered by one of the 3 perspectives). Overlaps in the coverage of constructs between AF, eHealth, and implementation indicate the integrative nature of the development and implementation of the APM-AF system. Overlapping constructs occurred more often for APM-AF systems (4/29, 14%) than for APM-AF development and implementation (2/35, 6%). In the former, most constructs (21/29, 72% and 20/29, 69%, respectively) came from AF and eHealth literature, whereas in the latter, most constructs (28/35, 80%) came from the implementation literature. Constructs underpinned by all 3 perspectives were not necessarily described by more studies (eg, goal setting; 5/12, 42%).

Comparison, Goal Setting, and Action Planning
Approximately 67% (8/12) of the studies described data comparison, either in terms of trends over time or benchmarks between groups and with other hospitals [52,57,59,60,63]. Approximately 33% (4/12) of the studies briefly mentioned goal setting and action planning. Goals were either derived from literature [52], based on current data [53,54], or described in terms of an HCW's need to discuss goals [59] but were not substantiated. Approximately 42% (5/12) of the studies mentioned action planning after feedback, but mostly in general terms (eg, feedback as a tool from which participants could make an actionable plan) [55,61,62], or as a separate follow-up strategy besides the APM-AF system, requiring additional information and human involvement [58,59,61,62].

Design
Design details about included graphs were described by 33% (4/12) of the studies [53,55,60,64], whereas 42% (5/12) of the studies broadly described the APM-AF system design [52,54,59,62,63]. One of the studies used a theory-informed design to ensure that their design matched task and user characteristics [55]. Approximately 33% (4/12) of the studies described an interactive and customizable AF system, wherein personalization was used to customize feedback to match end users' needs [55,59,60,64]. However, neither was this further specified (eg, which parts are customizable) nor did it focus on user-guided experience (ie, how usability is ensured).

APM-AF Development and Implementation Constructs (Research Question 4)
Overview Table 5 shows the APM-AF development and implementation constructs (Multimedia Appendix 4 [52-55,57-64] provides more details). Constructs could be categorized into four main codes-(1) stakeholders and roles, (2) target behavior and added value, (3) embedding in practice and (4) formative evaluation-and are elaborated upon below. Table 5 also shows to what degree and by which studies these constructs were described.

Inner and Outer Setting
Few studies (4/12, 33%) paid attention to the inner setting, expressing the need for a collaborative environment and open culture, in which the quality of their work can be discussed safely [59,61,63]. One of the studies described increased patient involvement as a result of visible APM-AF systems [54].

APM-AF System Use
Intended and actual use of the APM-AF system was hardly (3/12, 25%) described, either as the maximum time HCWs should spend on filling out the audit tool [64] or as how often and complete the audit tool was filled in [58]. In addition, one of the studies used Google Analytics to measure users' interactions with gamification parts [60].

Harms or Unintended Effects
Approximately 33% (4/12) of the studies described whether and how harms and unintended effects were monitored during the development and implementation process (in general terms or with specific examples) [54,61,63,64].

Replicability and Digital Preservation
One of the studies published their APM-AF system on the web with additional information (eg, web forum) [64].

Principal Findings
This scoping review aimed to provide insights into strategies and theoretical underpinnings for AF system development and implementation from a sociotechnical perspective. Of the 2125 studies found in the search, 12 (0.56%) studies were included describing the development and implementation of their AF systems heterogeneously in terms of study aims, AF targets, and development and implementation strategies. Approximately 17% (2/12) of the studies explicitly aimed to illustrate how TMFs could guide choices in AF system development and implementation. A comprehensive interdisciplinary conceptual framework, based on overlapping and complementary constructs from TMFs from AF, eHealth and interventions, and implementation literature, was created to compare the studies.

Lessons Learned for the Development and Implementation of APM-AF Systems (Research Question 5)
In this discussion, research question 5 is answered by providing lessons learned from reflecting upon our findings for theoretical underpinnings of and reporting on AF, AF systems, and their development and implementation.

Theoretical Underpinnings for AF
With health-and health care-related problems becoming increasingly complex, interdisciplinary theoretical integration to combine different perspectives is inevitable and pivotal to grasp the complexity of real-world problems [65]. This study showed the added value of considering and combining AF, eHealth, and implementation literature for studying AF systems. AF literature covered mostly AF system constructs (21/29, 72%), whereas relevant development and implementation constructs were hardly covered (12/35, 34%). Therefore, studies using only AF literature might miss important development and implementation constructs, such as stakeholder roles (eg, leadership engagement and champions) and embedding in practice (eg, implementation complexity and setting), that influence AF effectivity and efficiency [66,67]. Therefore, we argue that TMFs for AF and for development and implementation should be balanced, as exemplified in our interdisciplinary conceptual framework, and matched with studies' specific research objectives, methods, and context (eg, setting and participants) [68].
However, selecting and combining the best-fitting TMFs remains a challenge [69]. Well-known examples of combined frameworks exist in implementation science (eg, Theoretical Domains Framework [70]); however, little information is provided about how constructs were combined. Overall, there is little guidance on the selection and integration of interdisciplinary TMFs [71]. Evolving initiatives that create shared languages across disciplines and theories (eg, CohenMiller and Pate [72]) and provide criteria for theory selection (eg, Birken et al [73]) are encouraged.

Reporting on AF Systems
This scoping review resulted in an overview of constructs for APM-AF systems (Table 4), enriching the AF best practices from Colquhoun et al [18] with constructs of audit (eg, auditees and audit input), feedback framing and incentives, and AF system constructs (eg, technology and materials, human-system interaction, and data validity and trust and credibility). For replicability and using the framework in actual development and implementation, reporting about the audit input and what technology and materials were used is important. Furthermore, data validity and credibility was deemed as one of the 5 most important aspects for AF in a recent study [42].
In our view, the 2 constructs that were underreported in the included studies require attention in future studies. First, we observed quite broad descriptions of AF system design, with a lack of attention to functional (ie, what can the AF system do) and visual (ie, how efficiently and effectively information is presented to users) design, and engagement with the AF system. However, these constructs are important for how an AF system is received and used in practice [74]. The lack of design aspects and considerations of engagement might reflect the neglect of eHealth system characteristics (such as design) and engagement as active influencers for eHealth effectivity [75]. Second, comparison, goal setting, and action planning were hardly described in the included studies. A lack of substantiations for comparisons was also reported by a review on clinical performance comparators for AF on various clinical topics [76], whereas the underuse of goal setting and action planning was also found by a systematic review on behavior change interventions for APM in hospitals [28]. These 3 constructs were derived from all included theoretical perspectives and are common behavior change techniques [77], suggesting that they require and deserve more attention in future studies.

Reporting on Development and Implementation of AF Systems
Of the full-text studies assessed for eligibility, most studies (30/58, 52%) were excluded as they focused primarily on effectiveness and did not sufficiently report on development and implementation. The inclusion of 12 studies in full-text might seem little; however, we believe this is exemplary of the tendency in (APM and eHealth) research to publish more about effectiveness than about the development and implementation process [3,11,12]. Therefore, in future AF system studies, but also for other eHealth technologies, considering development and implementation as influencers of the effectivity and efficiency of eHealth in practice has yet to gain ground [13]. During the construction of the conceptual framework, the interwovenness of development, implementation, and formative evaluation became apparent through the many overlaps observed. This resonates with best practices in eHealth development and implementation, which state that implementation is integrated with development and requires an iterative and holistic approach [78]. Therefore, next to reporting on evaluation, studies should report on both the constructs for AF systems and for development and implementation.
There is no single right development and implementation approach because of the many variations in APM and AF objectives, stakeholders, technologies, and settings [79]. Therefore, the constructed conceptual framework should be seen as a checklist that provides general guidance for potentially interesting constructs to consider; it remains at the discretion of researchers and developers which and how constructs are incorporated in their study. At a minimum, we propose to reflect upon relevant stakeholders and their roles, implementation complexity, compatibility with target behavior, and work processes, including the added value of AF and formative evaluation. Overall, supporting researchers and HCWs in selecting and integrating TMFs into their development and implementation process and promoting the explicit reporting of the theoretical underpinning and substantiation of choices are highly encouraged [41].
The constructed conceptual framework can be used in future studies to ensure a comprehensive view of AF for the transparency and replicability of individual studies. Therefore, we recommend using the conceptual framework as a checklist and adding it (including substantiation of choices) as a supplementary material in future publications. Furthermore, findings from this study can be used to improve the professionalization and harmonization of AF studies, which is important considering the increasing use of AF principles in learning health care systems [80]. The lack of attention to factors that support the development of learning health care systems (eg, organizational culture and cooperation between HCWs and researchers) is recognized as an important barrier to the widespread adoption of learning health care systems [81]. As these aspects are included in the constructed conceptual framework, it might help future researchers and developers to explicitly consider and integrate these constructs in their AF or learning system.

Limitations and Strengths
This scoping review has several limitations. Although a comprehensive search query was used to search the most important databases for health research, we only included peer-reviewed published research and excluded evaluation studies. As a result, it might be possible that relevant findings were omitted (eg, from gray literature). Two systematic reviews on AF for various health targets were screened to ensure that no relevant publications were missed [1,2]. Another limitation is that evaluation studies were excluded from this review to highlight constructs relevant to the development and implementation, whereas evaluation is critical to know whether an intervention was successful. Therefore, it will be an important next step to evaluate AF systems in terms of processes (eg, improved HH), clinical outcomes (eg, reduced number of infections and decreased AMR), and technological outcomes (eg, APM-AF system use and persuasiveness) [82]. Finally, data extraction relied on the subjective interpretation of the constructs from the included publications by 1 researcher. However, the conceptual framework ( Table 2 and Table 3) provided a thorough base for systematic and structured assessment, and the findings were iteratively discussed and revised throughout the review process.

Conclusions
This scoping review provides novel insights into the theoretical underpinning of and reporting on the development and implementation of AF systems while demonstrating how a comprehensive conceptual framework can be created and used and a valuable means for capturing relevant constructs from heterogeneous studies with varying theoretical underpinnings. Few studies have explicitly described how choices for AF systems and their development and implementation processes were substantiated by TMFs. The interdisciplinary conceptual framework developed in this study is a first step toward the professionalization and harmonization of AF development and implementation with a sociotechnical approach. It provides guidance and a comprehensive checklist to guide researchers, HCWs, and policy makers in making informed choices in the development and implementation of AF systems, with the aim of further improving the quality and safety of health care.