JMIR J Med Internet Res Journal of Medical Internet Research 1438-8871 JMIR Publications Toronto, Canada v24i3e33531 35275082 10.2196/33531 Review Review Improving the Development and Implementation of Audit and Feedback Systems to Support Health Care Workers in Limiting Antimicrobial Resistance in the Hospital: Scoping Review Kukafka Rita Hirosawa Takanobu Laukka Elina Keizer Julia PhD 1
Centre for eHealth and Wellbeing Research Section of Psychology, Health and Technology University of Twente PO Box 217 Enschede, 7500 AE Netherlands 31 53 489 1289 j.keizer@utwente.nl
https://orcid.org/0000-0003-0080-436X Bente Britt E MSc 1 https://orcid.org/0000-0002-9142-4946 Al Naiemi Nashwan MD, PhD 2 3 https://orcid.org/0000-0002-0176-9853 Van Gemert-Pijnen Lisette JEWC PhD 1 https://orcid.org/0000-0001-6511-7240 Beerlage-De Jong Nienke PhD 4 https://orcid.org/0000-0002-1567-5196 Centre for eHealth and Wellbeing Research Section of Psychology, Health and Technology University of Twente Enschede Netherlands Laboratorium Microbiologie Twente Achterhoek Hengelo Netherlands Department of Infection Prevention Hospital Group Twente Almelo/Hengelo Netherlands Section of Health Technology and Services Research Technical Medical Centre University of Twente Enschede Netherlands Corresponding Author: Julia Keizer j.keizer@utwente.nl 3 2022 11 3 2022 24 3 e33531 24 9 2021 19 10 2021 1 12 2021 3 12 2021 ©Julia Keizer, Britt E Bente, Nashwan Al Naiemi, Lisette JEWC Van Gemert-Pijnen, Nienke Beerlage-De Jong. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 11.03.2022. 2022

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

 Background

For eHealth technologies in general and audit and feedback (AF) systems specifically, integrating interdisciplinary theoretical underpinnings is essential, as it increases the likelihood of achieving desired outcomes by ensuring a fit among eHealth technology, stakeholders, and their context. In addition, reporting on the development and implementation process of AF systems, including substantiations of choices, enables the identification of best practices and accumulation of knowledge across studies but is often not elaborated on in publications.

 Objective

This scoping review aims to provide insights into the development and implementation strategies for AF systems for a real-world problem that threatens modern health care—antimicrobial resistance—and provide an interdisciplinary conceptual framework that can serve as a checklist and guidance for making informed choices in the development and implementation of future AF systems.

 Methods

A scoping review was conducted by querying PubMed, Scopus, Web of Science, IEEE Xplore Digital Library, and Embase (≥2010) for studies describing either the development or implementation process, or both, of an AF system for antimicrobial resistance or infections in hospitals. Studies reporting only on effectiveness or impact were excluded. A total of 3 independent reviewers performed the study selection, and 2 reviewers constructed the conceptual framework through the axial and selective coding of often-used theories, models, and frameworks (TMFs) from the literature on AF and eHealth development and implementation. Subsequently, the conceptual framework was used for the systematic extraction and interpretation of the studies’ descriptions of AF systems and their development and implementation.

 Results

The search resulted in 2125 studies that were screened for eligibility, of which 12 (0.56%); 2012-2020) were included. These studies described the development and implementation processes heterogeneously in terms of study aims, study targets, target groups, methods, and theoretical underpinnings. Few studies have explicitly explained how choices for the development and implementation of AF systems were substantiated by the TMFs. The conceptual framework provided insights into what is reported on the development and implementation process and revealed underreported AF system constructs (eg, AF system design; engagement with the AF system; and comparison, goal setting, and action planning) and development and implementation (eg, champions) constructs.

 Conclusions

This scoping review showed the current heterogeneous reporting of AF systems and their development and implementation processes and exemplified how interdisciplinary TMFs can (and should) be balanced in a conceptual framework to capture relevant AF systems and development and implementation constructs. Thereby, it provides a concrete checklist and overall guidance that supports the professionalization and harmonization of AF system development and implementation. For the development and implementation of future AF systems and other eHealth technologies, researchers and health care workers should be supported in selecting and integrating TMFs into their development and implementation process and encouraged to explicitly report on theoretical underpinnings and the substantiation of choices.

 scoping review audit and feedback eHealth development implementation antimicrobial resistance antibiotic stewardship infection control Introduction Background

Audit and feedback (AF) is a common reflective approach for various health care targets; however, the reported effects are small to moderate [1]. With the increase in electronically available data in health care, there is great potential for electronic AF systems [2]. The effectiveness of AF systems depends on the targeted behavior and the content, delivery, and context of AF and the system [1,3-6]. These constructs are often studied after AF system development and implementation to evaluate strategies and their ingredients for success [7-10]. In the literature, less attention has been paid to the development and implementation processes of AF systems [3], as is also common in the broader field of eHealth [11,12]. The development process of eHealth can refer to the entire iterative process of developing an eHealth technology, from predesign and design to implementation and (summative) evaluation [13]. However, in this study, we focus on the process from predesign and design (referred to hereafter as development) to implementation, including formative evaluation cycles. This allows us to focus on the early stages of implementation and development that are truly intertwined, as potential implementation issues (eg, limited eHealth skills) should be accounted for early in the development process to avoid well-known pitfalls of stakeholder and context disregard [14]. These phases are entwined by iterative formative evaluation cycles that provide ongoing information on how to improve both the eHealth technology and the development process taking [13].

Development and implementation are essential to gain a profound understanding of relevant stakeholders, their thinking and work processes, and their context (including implementation factors). Without this understanding, a misfit among technology, context, and people is likely to occur, which decreases the effectivity and efficiency of eHealth in practice [13]. It is crucial to consider these constructs from the start of the development and implementation process to avoid common pitfalls in current AF, such as top-down expert-driven audits with feedback at the hospital level rather than personalized, actionable feedback that supports health care workers (HCWs) in improving the quality and safety of health care [15,16].

The application of theories, models, and frameworks (TMFs) is advocated as an integral part of eHealth development and implementation as it identifies what works for whom, why, how, and when, likely resulting in eHealth technology that achieves the desired outcomes [17]. Colquhoun et al [18] and Tuti et al [2] reported that only 9% (n=140) and 29% (n=7) of the included studies in their systematic reviews explicitly used theory to inform AF development and design. Therefore, implicit assumptions about AF working mechanisms and effective AF systems have driven AF development. Although these assumptions might hold true, they were not informed by theory [18,19], whereas there is a clear link between TMFs and eHealth intervention effectiveness [20,21].

To study the development and implementation of AF, this scoping review focuses on a real-world, wicked problem—antimicrobial resistance (AMR). AMR poses an increasing threat to human health and the durability of modern health care [22]. By 2050, AMR is expected to cause more yearly deaths worldwide than cancer currently does [23]. Antimicrobial and diagnostic stewardship programs and infection control programs form an integrated approach of AMR prevention measures (APMs) that aim to reduce and prevent the burden of AMR in hospitals [24]. Previous studies on HCWs’ needs for APM support showed that changing HCWs’ beliefs about their contribution to limiting AMR should be an important aim of APM strategies rather than merely focusing on raising AMR awareness or influencing ad hoc decisions [25,26]. To do so, learning through reflective cycles provides the opportunity to change HCWs’ behaviors by giving them insight into their own behavior and improvement possibilities [15,27]. Therefore, AF for APM (APM-AF) is a promising strategy in the fight against AMR, although it is currently underused and understudied in the field of AMR [7].

There is a clear link between the use of TMFs and APM effectiveness [28-31], and because of the interdisciplinary nature of APM and eHealth, approaches for development and implementation are grounded in a plethora of TMFs [32]. In particular, APM-AF combines behavior change techniques [28-31], participatory eHealth development [33], human-centered and persuasive design [34-37], and improvement [38] and implementation [39] science. Moreover, TMFs have emerged for AF itself (eg, actionable feedback and feedback intervention theory [3-6]) and in the field of AMR (eg, integrated stewardship model [16,24,40]). Combining these TMFs into a conceptual framework that guides the development and implementation of APM-AF is challenging, and there is little guidance on how to create such interdisciplinary conceptual frameworks [41,42].

 Objectives

There seems to be no standardized way of (theoretically) substantiating choices for and reporting on the development and implementation of AF systems, which hinders the identification of best practices and knowledge accumulation [10,43]. Whereas other reviews on AF have mainly focused on the effectiveness of AF systems [1,2], this scoping review focuses on the development and implementation process to further harmonize and professionalize AF system development and implementation. The aim of this study is to gain insight into the development and implementation strategies for APM-AF systems by answering the following research questions:

What studies have been conducted so far to study the development and implementation of APM-AF systems?

What TMFs are used and described in studies on the development and implementation processes of APM-AF systems?

What information has been reported on APM-AF systems, and how are choices substantiated?

What information has been reported on the development and implementation processes of APM-AF systems, and how are choices substantiated?

What are the lessons learned for the development and implementation of APM-AF systems?

To allow for an evidence synthesis of information on the development and implementation of APM-AF, and because of the explorative aim and research questions in this study, a scoping review is preferred over a systematic literature review [44,45]. This scoping review provides an interdisciplinary conceptual framework that supports researchers, HCWs, and policy makers to make informed choices in APM-AF system development and implementation, with the aim of reducing the burden of AMR and improving the quality and safety of health care.

 Methods

The PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist was used to report on this scoping review without a prior registered review protocol [46]. This scoping review was designed by a multidisciplinary research team comprising AMR and eHealth experts.

Eligibility Criteria

Studies were included if (1) they described the development and implementation process of an AF system (including monitoring and surveillance systems), (2) the system provided feedback to HCWs, and (3) the system targeted AMR and infections in hospitals. A more elaborate description of development and implementation is provided in Multimedia Appendix 1 [13]. We define AF systems as any system that comprises AF, wherein at least one of them (audit or feedback) is delivered or enhanced through the internet and related technologies [47]. As reporting on eHealth development and implementation processes is highly heterogeneous, there were no requirements for specific TMFs, methods, or eHealth technologies. Reviews and poster abstracts were excluded, as were studies outside the hospital setting. Evaluation studies that only reported on APM-AF effectiveness and impact without reporting on development and implementation were excluded. However, constructs of formative evaluation (defined as “activities throughout the entire development process that provide ongoing information on how to improve the development process, outcomes of activities and eHealth technology” [13]) were included, as it is intertwined throughout the eHealth development and implementation process. A full list of eligibility criteria can be found in Multimedia Appendix 1.

 Information Sources, Search, and Selection of Evidence

A comprehensive and systematic literature search in PubMed, Scopus, Web of Science, IEEE Xplore Digital Library, and Embase was conducted without language restrictions. Only studies published in and after 2010 were considered, as both eHealth development and implementation and AMR and APM are rapidly advancing fields. Databases were queried by JK on November 2, 2020, except for Embase, which was queried on January 28, 2021. Together with an information specialist, AMR experts, and eHealth researchers, a structured query was constructed comprising the following terms: audit OR monitor OR surveillance AND feedback AND develop* OR implement* AND infection OR antib* OR antimicrobial OR resistance. The results were uploaded to the Covidence web-based software platform (Veritas Health Innovation Ltd), where duplicates were removed. Sources of evidence were selected in a thorough screening process, including title and abstract screening and full-text screening by three researchers independently (JK, BB, and NBJ). After each round, conflicts were discussed until a consensus was reached.

 Data Charting Process

To chart the data, JK created a data extraction form (Multimedia Appendix 2 [2-5,18,43,48-50]) in Microsoft Excel. The general study characteristics extracted were first author, year, journal, country, study aims, targets and target groups, study design and methods, and theoretical underpinning. Given the heterogeneous study approaches and theoretical underpinnings of the included studies, a comprehensive overarching conceptual framework was needed to systematically analyze relevant constructs. The conceptual framework was grounded in often-used TMFs and best practices from various scientific perspectives on AF [3-5,18] and for the description, development, and implementation of eHealth interventions in general [2,43,48-50]. These TMFs and best practices were merged via an iterative axial and selective coding process by JK and NBJ. Thereby, overlapping and complementary constructs from various scientific perspectives were revealed. To structure all constructs, a distinction was made between constructs for APM-AF systems (n=41; research question 3) and constructs for development and implementation (n=35; research question 4).

The data extraction form was discussed within the research team, piloted, and iteratively refined throughout the assessment process. Note that this conceptual framework should be merely seen as an analytic framework for an organized way of thinking about and reporting on APM-AF systems from various perspectives and not as a theory explaining or predicting possible interrelations and outcomes.

 Synthesis of Results

The main researcher (JK) read all full texts and systematically extracted the data using the data extraction form. Studies were scored with a + for a comprehensive, ~ for an incomplete, and - for a missing description for each data field. Descriptions were copied from the studies and further summarized per data field by describing variations among studies (ie, axial coding). In this process, data fields described by none of the studies were omitted (Multimedia Appendix 2), and other overlapping fields were combined. This reduced the number of data fields for APM-AF systems to 29. The translation and summarization of the extracted data into results were discussed in various rounds within the research team. Owing to the heterogeneity and qualitative nature of the included study designs, the richness and relevance of the (contextual) information were believed to be more important than study quality. Therefore, no quality appraisal was performed [51].

 Results Study Selection

The literature search resulted in 3592 potentially relevant abstracts. Of the 3592 papers, after removing 1467 (40.84%) duplicates, 2125 (59.16%) unique titles and abstracts were assessed (Figure 1), which resulted in the eligibility assessment of 58 (1.61%) full texts. The main reasons for exclusion were a lack of information on development or implementation and evaluation studies (without reporting on development or implementation).

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart of included and excluded studies, including reasons for exclusion.

 Current State of the Literature Addressing APM-AF Development and Implementation (Research Question 1) Study Characteristics

In total, of the 58 papers, 12 (21%) were included in this review (2012-2020), mostly from PubMed, Scopus, and Web of Science. Publications came from Northern American (6/12, 50%) or European (4/12, 33%) countries and Australia (2/12, 17%). Included studies stemmed from journals in various research fields (eg, infections or implementation science). Studies described APM-AF systems that were either in (preparation of) development or already implemented in practice, resulting in a wide variety of study aims, study targets, target groups, study designs, and used methods (Table 1).

Study characteristics.

Author and country Journal Study aims Study targets Target group Study design and methods Theoretical underpinning
Boscart et al, Canada [52] Implementation science To identify nurses’ and administrators’ perceived barriers and facilitators to current HHa practices and the implementation of a new electronic monitoring technology for HH ICPb and HAIc: HH (improving HH compliance) Nurses and administrators Qualitative:

Semistructured key informant interviews

 Theoretical Domains Framework
Conway et al, United States [53] The Joint Commission Journal on Quality and Patient Safety To describe the implementation of an automated group monitoring and feedback system for promoting HH among HCWsd and report its impact on the frequency of HH at a community hospital ICP and HAI: HH (increase HH frequency) HCWs (eg, nurses and respiratory therapists), administrators, and managers Multiple methods:

Quantitative: before-and-after study on HH events per patient hour (outcome)

Qualitative: focus groups

 Model of Actionable Feedback
Edmisten et al, United States [54] American Journal of Infection Control To describe the implementation of an electronic HH monitoring system in 3 community hospitals, including the challenges and drivers of success and the maintenance activities needed for continued improvements in compliance with HH practices ICP and HAI: HH (improving HH compliance) HCWs, staff, unit/department directors and, facility management Multiple methods:

Quantitative: after study (outcome measures on HH compliance after implementation)

Qualitative: direct input from users/department and facility leaders, direct observation, and analysis of system-generated data and sharing of best practices between facilities

 None reported
Hysong et al, United States [55] BMJ Quality and Safety To describe how feedback intervention theory can be systematically applied in health care settings to design better feedback interventions DSPe and HAI: to improve internal-medicine resident’s and long-term care personnel’s capacity to distinguish between asymptomatic bacteriuria and catheter-associated urinary tract infection HCWs (eg, nurse practitioners and staff physicians) Multiple methods:

Quantitative: the Smither et al [56] 11-item scale for recipients’ reactions to feedback

Quantitative: chart monitoring (adherence to the treatment algorithm, specifically, rates of urine culture) of orders and inappropriate use of antibiotics

 Feedback Intervention Theory
James, Australia [57] The Journal of Antimicrobial Chemotherapy To design an audit tool that was appropriate for use in all Australian hospitals, suited to local user requirements, and included an assessment of the overall appropriateness of the prescription ASPf: to improve the quality of patient care by reducing inappropriate and unnecessary use of antimicrobials (national level focus) HCWs (eg, pharmacists and nurses) Multiple methods:

Quantitative: interrater reliability and validity tests and web-based questionnaire

Qualitative: teleconference and direct input from users

 None reported
Jeanes et al, United Kingdom [58] American Journal of Infection Control To develop and implement an infection control performance and quality improvement data collection tool to meet the needs of large, acute health care providers and improve the credibility and use of infection control performance monitoring ICP and HAI: to improve the credibility and use of infection control performance monitoring (beyond HH) Not clearly described; (“auditors” and managers) Multiple methods:

Quantitative: questionnaires and intermittent validation

Qualitative: day to day contacts with auditors, feedback from users via the IC-CQIg data input system, discussion groups, and IC-CQI training sessions

 Pronovost Knowledge Translation Cycle and Barriers and Mitigation tool, double loop learning cycle, and Hexagon tool framework
Keizer, the Netherlands [59] Lecture Notes in Computer Science To describe how a bottom-up participatory development approach can improve the persuasive design of data-driven technologies for their end user (ie, HCWs) and within their context and describe how bottom-up participatory development is a necessary precondition for the development of persuasive data-driven technologies that foster sustainable implementation DSP, ASP, and ICP: to optimize HCWs’ diagnostic, antibiotic prescription and infection control behavior to limit AMRh HCWs (eg, urologists and residents) Multiple methods:

Quantitative: questionnaire

Qualitative: 2 focus groups (last focus group prototype based)

 CeHResi road map
Marques, Portugal [60] BMC Medical Informatics and Decision Making To develop a gamification solution that can provide HCWs real-time feedback on personal HH compliance ICP and HAI: to create awareness regarding HCWs’ HH compliance while trying to change their behaviors and optimize their performance Nurses Multiple methods (2 work iterations):

Qualitative: preliminary experiments, simulations, field studies, and focus groups

 Design Science Research Methodology and gamification
Pakyz, United States [61] American Journal of Infection Control To identify the factors related to the implementation of ASP strategies ASP: to optimize the use of antimicrobial agents, decrease AMR, and decrease rates of Clostridium difficile infection ASP pharmacists and physicians Multiple methods:

Quantitative: survey

Qualitative: semistructured telephone interviews

 None reported
Parker, Australia [62] Journal of Clinical Nursing To provide insights into the experiences of clinicians in implementing a multifaceted bundled urinary catheter care intervention (of which AFj is a considerable component) HAI: the study aimed to reduce catheter use and duration of catheterization Clinicians (eg, nurses and resident medical officers) Qualitative:

Postimplementation focus groups

 Intervention Description and Replication framework
Patel, United States [63] Interdisciplinary Perspectives on Infectious Diseases To describe the development and implementation of their AF intervention using a theoretical framework ASP: to promote the judicious use of antibiotics HCWs (eg, neonatologists and pediatric residents) Multiple methods:

Quantitative: retrospective observational study of antibiotic use and clinical vignette study

Qualitative: ethnographic workflow study and 2 focus groups

 Model of Actionable Feedback
Power, United Kingdom [64] International Journal for Quality in Health Care To set up a low-cost pragmatic system to provide monthly data on 4 harms across care settings and produce measures that could be used locally for improvement but also aggregated to determine the burden of harm nationally HAI: to reduce 4 high volume harms (safety outcomes), pressure ulcers, falls, urinary tract infection in patients with catheters, and venous thromboembolism HCWs (eg, nurses and junior physicians) Multiple methods:

Quantitative: questionnaire survey (professional satisfaction)

Qualitative: paper-based prototyping, formative evaluation by interaction with testers, web forum (including mail queries), regional leads, face-to-face meetings, and regional measurement workshops

 ProjectPplan Framework and Plan, Do, Study, Act Method

aHH: hand hygiene.

bICP: infection control program.

cHAI: hospital-acquired infection.

dHCW: health care worker.

eDSP: diagnostic stewardship program.

fASP: antimicrobial stewardship program.

gIC-CQI: Infection Control Continuous Quality Improvement.

hAMR: antimicrobial resistance.

iCeHRes: Center for eHealth Research.

jAF: audit and feedback.

 Study Aims

Of the 12 studies, 4 (33%) primarily focused on development, 4 (33%) on implementation, and 4 (33%) described both development and implementation. However, development and implementation appeared to be undefined concepts, with implementation studies describing the development and design constructs and development studies paying attention to implementation constructs. Studies aimed at describing APM-AF system development focusing on (1) the integration of TMFs (eg, Feedback Intervention Theory), (2) AF content and presentation (eg, feedback gamification), or (3) technical aspects (eg, suitable badges for hand hygiene [HH] monitoring). In addition, studies focused on implementation barriers and facilitators.

 Study Targets and Target Groups

Of the 12 studies, 11 (92%) focused on one of the APM (ie, diagnostic stewardship programs, antimicrobial stewardship programs, or infection control programs), whereas 1 (8%) targeted multiple APM. The target groups comprised a variety of HCWs (both frontline staff and AMR experts; 8/12, 67%) and in some studies, administrators and managers (4/12, 33%) as well.

 Study Design and Methods

Most studies (10/12, 83%) used multiple methods, complementing quantitative (eg, questionnaires) with qualitative data (eg, observations, interviews, and focus groups). Approximately 17% (2/12) of the studies were fully qualitative, relying on interviews and focus groups.

 TMFs for APM-AF Development and Implementation (Research Question 2) Theoretical Underpinning Described by Studies

Most studies (9/12, 75%) described the theoretical underpinnings of their APM-AF system or study approach (Table 1). Approximately 17% (2/12) of the studies explicitly mentioned the use of Feedback Intervention Theory and the Model of Actionable Feedback to guide choices in the development and implementation of their study aims [55,63], whereas others mentioned TMFs in their Introduction or Methods section. AF TMFs (3/12, 25%; eg, Model of Actionable Feedback) [53,55,63] were used, as were TMFs, for developing, implementing, and evaluating interventions or technologies (5/12, 42%; eg, Center for eHealth Research road map) [58-60,62,64] and for identifying implementation barriers/facilitators (1/12, 8%; eg, Theoretical Domains Framework) [52]. Substantiations of choices on APM-AF systems were scarce; few studies substantiated their choices, which were either theory informed (eg, providing group-level feedback) or based on findings from the studies themselves (ie, revisions based on formative evaluation).

 Conceptual Framework for APM-AF Development and Implementation

The conceptual framework, which is based on often-used TMFs and best practices for AF and eHealth interventions, is shown in Table 2 (APM-AF system constructs) and Table 3 (development and implementation constructs) and in Multimedia Appendix 2 in more detail.

Conceptual framework: APM-AFa system constructs (N=12)b.

Constructs and subconstructs Audit and feedbackc eHealth and interventionsd Implementatione Studies, n (%)
[18] [3] [5] [4] [48] [43] [2] [49] [50]
Audit
Auditeesf ✓ ✓ 10 (83)
Main inputb ✓ 9 (75)
Feedback
Feedback recipientsf ✓ ✓ ✓ 8 (67)
Main outputf ✓ ✓ ✓ 8 (67)
Level of individualization and specificity
Feedback provided to individual, groups, or bothf ✓ ✓ ✓ ✓ 11 (92)
Feedback is about the individual or team’s own behaviorsb ✓ ✓ ✓ ✓ 10 (83)
Feedback level specificityf ✓ ✓ ✓ ✓ 8 (67)
Comparison, goal setting, and action planning
Comparisonf ✓ ✓ ✓ ✓ ✓ 8 (67)
Goal settingg ✓ ✓ ✓ ✓ ✓ 5 (42)
Action planningg ✓ ✓ ✓ ✓ ✓ ✓ 4 (33)
Feedback framing and incentives
Punitivenessb ✓ ✓ 6 (50)
Attack on self-identityf ✓ ✓ 4 (33)
Intrinsic and extrinsic reinforcement or incentivesf ✓ ✓ 4 (33)
Timing
Delivery timingf ✓ ✓ ✓ ✓ 8 (67)
Timeliness (frequency and continuous cycle)f ✓ ✓ ✓ ✓ ✓ ✓ ✓ 11 (92)
Remindersf ✓ ✓ 3 (25)
APM-AF system
Technology and materials
Key features of the technologyf ✓ ✓ ✓ 11 (92)
Accessb ✓ 12 (100)
Materialsb ✓ ✓ ✓ 8 (67)
Human–system interaction
Modes of feedback deliveryf ✓ ✓ ✓ ✓ ✓ ✓ 9 (75)
Level of human involvementf ✓ ✓ 9 (75)
Engagementb ✓ 6 (50)
Design
Visual presentation strategies and cognitive loadg ✓ ✓ ✓ ✓ ✓ 9 (75)
User-guided experienceg ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 4 (33)
Data validity, d trust and credibility
Data validityb ✓ ✓ 9 (75)
Trust and credibilityf ✓ ✓ ✓ 11 (92)
APM-AF as learning strategy
Learning opportunities
Reflective learningf ✓ ✓ ✓ 5 (42)
Learning climatef ✓ ✓ 7 (58)
Additional strategies or proceduresb ✓ ✓ ✓ 12 (100)

aAPM-AF: audit and feedback for antimicrobial resistance prevention measures.

bUnique constructs (ie, where the various perspectives complement each other).

cApproximately 72% of constructs theoretically underpinned by literature on audit and feedback.

dApproximately 66% of constructs theoretically underpinned by literature on eHealth and interventions.

eApproximately 41% of constructs theoretically underpinned by literature on implementation.

fOverlapping constructs (constructs represented in 2 perspectives).

gOverlapping constructs (constructs represented in all perspectives).

Conceptual framework: APM-AFa development and implementation constructs (N=12)b.

Constructs and subconstructs Audit and feedbackc eHealth and interventionsd Implementatione Studies, n (%)
[18] [3] [5] [4] [48] [43] [2] [49] [50]
Stakeholders and roles
Developer or research teamb ✓ 5 (42)
Pilot testers and involvement in development and implementation processf ✓ ✓ ✓ 11 (92)
Leadership engagementb ✓ ✓ 6 (50)
Opinion leadersb ✓ 3 (25)
Formally appointed internal implementation leadersb ✓ 2 (17)
Championsb ✓ 3 (25)
Target behavior and added value
Target behavior, problem, and intervention
Nature of the problemb ✓ 12 (100)
Description of underlying behavior and decision processesb ✓ ✓ 12 (100)
Relevant sociocultural factors and comorbiditiesg ✓ 8 (67)
Perceived need for behavior changeg ✓ ✓ 4 (33)
Targeted behavior is likely to be amenable to feedbackb ✓ ✓ ✓ 6 (50)
Self-efficacyg ✓ ✓ ✓ 3 (25)
Justify need for behavior changeg ✓ ✓ ✓ 10 (83)
Rationale and added value
Rationale for using APM-AFg ✓ ✓ ✓ 12 (100)
Desirability, efficacy, safety, and cost effectivenessg ✓ ✓ ✓ 10 (83)
Relative advantageb ✓ 10 (83)
Embedding in practice
Implementation complexity and compatibility with target behavior and work processes
Complexity of implementation processb ✓ ✓ 8 (67)
Technology supply modelb ✓ 8 (67)
Compatibilityg ✓ ✓ ✓ 11 (92)
Remove barriersg ✓ ✓ ✓ 11 (92)
Opportunity costs (including additional efforts to use technology) g ✓ ✓ ✓ 3 (25)
Available resourcesb ✓ ✓ 6 (50)
Inner and outer setting
Structural characteristicsb ✓ ✓ 1 (8)
Networks and communicationsb ✓ ✓ 2 (17)
Cultureb ✓ 3 (25)
Patient needs and resourcesb ✓ 1 (8)
Implementation
Planningb ✓ ✓ 6 (50)
Executingb ✓ ✓ 5 (42)
Formative evaluation
Intended useb ✓ ✓ 1 (8)
Actual useb ✓ 3 (25)
Development process and formative evaluationsg ✓ ✓ 12 (100)
Harms or unintended effectsb ✓ 4 (33)
Trialabilityb ✓ 9 (75)
Revisions and updatingg ✓ ✓ ✓ 6 (50)
Replicability and digital preservationb ✓ 1 (8)

aAPM-AF: audit and feedback for antimicrobial resistance prevention measures.

bUnique constructs (ie, where the various perspectives complement each other).

cApproximately 32% of constructs theoretically underpinned by literature on audit and feedback.

dApproximately 24% of constructs theoretically underpinned by literature on eHealth and interventions.

eApproximately 74% of constructs theoretically underpinned by literature on implementation.

fOverlapping constructs (constructs represented in all perspectives).

gOverlapping constructs (constructs represented in 2 perspectives).

The construction of the comprehensive overarching conceptual framework revealed the added value of including multiple perspectives, as 48% of constructs were complementary (ie, covered by one of the 3 perspectives). Overlaps in the coverage of constructs between AF, eHealth, and implementation indicate the integrative nature of the development and implementation of the APM-AF system. Overlapping constructs occurred more often for APM-AF systems (4/29, 14%) than for APM-AF development and implementation (2/35, 6%). In the former, most constructs (21/29, 72% and 20/29, 69%, respectively) came from AF and eHealth literature, whereas in the latter, most constructs (28/35, 80%) came from the implementation literature. Constructs underpinned by all 3 perspectives were not necessarily described by more studies (eg, goal setting; 5/12, 42%).

 APM-AF System Constructs (Research Question 3) Overview

Table 4 shows APM-AF system constructs (more details in Multimedia Appendix 3 [52-55,57-64]). Constructs could be categorized into four main codes—(1) audit, (2) feedback, (3) APM-AF system, and (4) APM-AF as a learning strategy—and are elaborated upon below. Table 4 also shows to what degree and by which studies these constructs were described.

Constructs of APM-AFa systems (N=12).

Constructs and subconstructs Described by studies, n (%) References
Described elaborately and often substantiated Partially described or constructed without elaboration or substantiation Not described or substantiated
Audit
Auditees 10 (83) 1 (8) 1 (8) [52-55,57,59,60,62-64]
Main input 9 (75) 3 (25) 0 (0) [52-55,57,60,62-64]
Feedback
Feedback recipients 8 (67) 3 (25) 1 (8) [53-55,57,59,60,62-64]
Main output 8 (67) 3 (25) 1 (8) [53-55,57-60,62,63]
Level of individualization and specificity
Feedback provided to individual, groups, or both 11 (92) 1 (8) 0 (0) [52-55,57,59-64]
Feedback about the individual or team’s own behaviors 10 (83) 2 (17) 0 (0) [52-55,59-64]
Specificity 8 (67) 1 (8) 3 (25) [55,59,61,62,64]
Comparison, goal setting and action planning
Comparison 8 (67) 0 (0) 4 (33) [52,53,55,57,59,60,63,64]
Goal setting 5 (42) 1 (8) 6 (50) [52-54,59]
Action planning 4 (33) 3 (25) 5 (42) [55,58,59,61,62]
Framing and incentives
Punitiveness 6 (50) 0 (0) 6 (50) [53,55,59-61,63]
Attack on self-identity and cognitive influences 4 (33) 0 (0) 8 (67) [53,58,61,63]
Intrinsic and extrinsic reinforcement or incentives 4 (33) 0 (0) 8 (67) [52,55,60,61]
Timing
Delivery timing 8 (67) 0 (0) 4 (33) [52,54,60,61,64]
Timeliness 11 (92) 1 (8) 0 (0) [52-55,57-60,62,63]
Reminders 3 (25) 0 (0) 9 (75) [52,53,59]
APM-AF system
Technology and materials
Materials 11 (92) 1 (8) 0 (0) [52-55,57-60,62-64]
Key features of the technology 12 (100) 0 (0) 0 (0) [52-55,57-64]
Access 8 (67) 0 (0) 4 (33) [53-55,59,60,62-64]
Human–system interaction
Modes of feedback delivery 9 (75) 2 (17) 1 (8) [52,53,55,59-64]
Level of human involvement 9 (75) 3 (25) 0 (0) [53,55,57-59,61-64]
Engagement 6 (50) 0 (0) 6 (50) [55,59,60,62-64]
Design
Presentation strategies and cognitive load 9 (75) 1 (8) 2 (17) [52-55,59,60,62-64]
User-guided experience 4 (33) 3 (25) 5 (42) [55,59,60,64]
Validity and credibility
Data validity 9 (75) 1 (8) 2 (17) [52-55,57,58,60,61,64]
Trust and credibility 11 (92) 0 (0) 1 (8) [52,53,55,57-64]
APM-AF as a learning strategy
Learning opportunities
Reflective learning 5 (42) 0 (0) 7 (58) [55,59-61]
Learning climate 7 (58) 0 (0) 5 (42) [55,58,59,61,62,64]
Additional strategies or procedures 12 (100) 0 (0) 0 (0) [52-55,57-64]

aAPM-AF: audit and feedback for antimicrobial resistance prevention measures.

 Audit Auditees

The ones audited, or auditees, were described by most studies (10/12, 83%) and comprised frontline HCWs [52-55,57,59,60,62-64].

 Main Input

Approximately 42% (5/12) APM-AF systems relied on automatically collected input (eg, electronic HH monitoring system) [52-54,60,63], whereas 33% (4/12) systems relied on manual input (eg, audit survey tool) [55,57,62,64].

 Feedback Feedback Recipients

Feedback recipients were described by most studies (8/12, 67%) and comprised auditees (ie, frontline HCWs) [55,60,62] and managers or administrators [53,54,59,63,64].

 Main Output

Approximately 67% (8/12) of the studies described APM-AF output in terms of specific content (eg, process, structure, and outcome indicators) [53,54,58,60,62] or provided a general description of output (eg, quality of antibiotic treatment) [55,59,63].

 Level of Individualization and Specificity

The level of feedback individualization was described by most studies (11/12, 92%). Feedback was provided on individual [55,61] or group level [53,62-64] or on both (ie, option to choose) [52,54,57,59,60]. Approximately 8% (2/12) of the studies explicitly justified their choice to provide group-level feedback only as individual feedback could be perceived as too threatening [53,63]. Feedback was provided to the auditees only [55,60-62], to the auditees and managers or administrators [52,54,59,63,64], or to managers or administrators only [53]. Feedback specificity was described by most studies (8/12, 67%). Feedback was provided on individual patient cases (mostly diagnostic and antimicrobial stewardship studies) [55,61], on aggregated group level (mostly infection control studies) [53,54,60], or on both individual and aggregated levels [59,62,64].

 Comparison, Goal Setting, and Action Planning

Approximately 67% (8/12) of the studies described data comparison, either in terms of trends over time or benchmarks between groups and with other hospitals [52,57,59,60,63]. Approximately 33% (4/12) of the studies briefly mentioned goal setting and action planning. Goals were either derived from literature [52], based on current data [53,54], or described in terms of an HCW’s need to discuss goals [59] but were not substantiated. Approximately 42% (5/12) of the studies mentioned action planning after feedback, but mostly in general terms (eg, feedback as a tool from which participants could make an actionable plan) [55,61,62], or as a separate follow-up strategy besides the APM-AF system, requiring additional information and human involvement [58,59,61,62].

 Feedback Framing and Incentives

Some studies mentioned feedback framing in terms of punitiveness (6/12, 50%) or an attack on self-identity (3/12, 25%); however, few specified whether and how these constructs were incorporated in AF system design [53,55,58-61,63]. Approximately 8% (2/12) of the studies incorporated these constructs in their decision to focus only on group-level feedback [53,63], whereas 17% (2/12) studies emphasized nonconfrontational and informal language [60,61]. Intrinsic and extrinsic reinforcement and incentives were addressed in general terms [52,55,61] and more specifically by 8% (1/12) of the studies (eg, competition, win state, and rewards), including how these were implemented in the system (eg, presenting the highest score with a distinct color) [60].

 Timing

Approximately 42% (5/12) of APM-AF systems made use of feedback at the point of care. This was provided via visual and auditory signals (for HH monitoring) [52,54] or a real-time performance dashboard [54,60,64]. Retrospective feedback was provided in 10 systems, with variable frequencies (daily, monthly, half yearly, and yearly) [52-55,57-60,62,63], sometimes with the possibility of continuously accessing the performance dashboard when needed [53,59,60]. Reminders were mentioned in 25% (3/12) of the studies [52,53,59].

 APM-AF System Technology and Materials

All studies described their (envisioned future) technologies, which ranged from audit tools (eg, Microsoft Excel or SurveyMonkey [Momentive Inc]) [57,58,61,62,64] to electronic monitoring systems (for HH) [52-54,60], interactive Microsoft PowerPoint presentations [55,63], and prototypes [59]. Access to the APM-AF systems was realized in interactive dashboards or Microsoft PowerPoint presentations with the possibility of sending customized reports via email [54,59,60,64], whereas, in 33% (4/12) of the studies, feedback recipients relied on managers or the research team (via email or face-to-face) for access to the APM-AF system [53,55,62,63].

 Human–System Interaction

Approximately 17% (2/12) of the studies solely provided feedback via the APM-AF system [60,64], whereas 58% (7/12) of the studies also provided face-to-face feedback [52,53,55,59,61-63]. In addition, studies described the need for additional human involvement, for example, for (educational) meetings with AMR experts [53,57,59,62,63], and support in data processing [55,58,59,61,64]. Half of the studies described how they would engage the user with the APM-AF system via interactive feedback presentations [55,64], gamification [60], or with additional strategies (eg, creating an AF task force) [59,62,63].

 Design

Design details about included graphs were described by 33% (4/12) of the studies [53,55,60,64], whereas 42% (5/12) of the studies broadly described the APM-AF system design [52,54,59,62,63]. One of the studies used a theory-informed design to ensure that their design matched task and user characteristics [55]. Approximately 33% (4/12) of the studies described an interactive and customizable AF system, wherein personalization was used to customize feedback to match end users’ needs [55,59,60,64]. However, neither was this further specified (eg, which parts are customizable) nor did it focus on user-guided experience (ie, how usability is ensured).

 Validity and Credibility

Data validity was addressed in terms of raised concerns by study participants [52-54,61], (planned) data validation activities [55,57,58,64], or technical constructs [60]. The trust in and credibility of the APM-AF system was addressed by describing study participants’ perceptions [52,53,58-62] or (planned) activities to improve the trust in and credibility of the system [55,57,63,64].

 APM-AF as a Learning Strategy Learning Opportunities

Approximately 33% (4/12) of the studies described reflective learning (ie, personal reflections on one’s behavior and APM performance) as a result and strength of APM-AF systems [55,59-61]. Furthermore, APM-AF systems (6/12, 50%) were described as a (potential) facilitator for interactive discussions and communication between HCWs and AMR experts, mostly in existing meetings [55,58,59,61,62,64].

 Additional Strategies or Procedures

All studies described additional strategies, both for the study (eg, webinar to explain how to use the tool) and for the APM-AF system in practice (eg, creating an AF task force) [52-55,57-64].

 APM-AF Development and Implementation Constructs (Research Question 4) Overview

Table 5 shows the APM-AF development and implementation constructs (Multimedia Appendix 4 [52-55,57-64] provides more details). Constructs could be categorized into four main codes—(1) stakeholders and roles, (2) target behavior and added value, (3) embedding in practice and (4) formative evaluation—and are elaborated upon below. Table 5 also shows to what degree and by which studies these constructs were described.

APM-AFa development and implementation constructs.

Constructs and subconstructs Described by studies, n (%) References
Described elaborately and often substantiated Partially described or constructed without elaboration or substantiation Not described or substantiated
Stakeholders and roles
Stakeholders
Developer or research team 5 (42) 4 (33) 3 (25) [53,55,57,59,61]
Pilot testers and involvement in development and implementation process 11 (92) 2 (17) 1 (8) [52-55,57,59-64]
Leadership engagement 6 (50) 2 (17) 4 (33) [54,58,61-63]
Opinion leaders 3 (25) 0 (0) 9 (75) [58,61,62]
Formally appointed internal implementation leaders 2 (17) 0 (0) 10 (83) [61,62]
Champions 3 (25) 1 (8) 8 (67) [52,61,62]
Target behavior and added value
Target behavior, problem, and intervention
Description of underlying behavior and decision processes 8 (67) 2 (17) 2 (17) [52,53,55,57-59,62,63]
Nature of the problem 12 (100) 0 (0) 0 (0) [52-55,57-64]
Relevant sociocultural factors and comorbidities 12 (100) 0 (0) 0 (0) [52-55,57-64]
Tension for behavior change 4 (33) 1 (8) 7 (58) [52,53,55,60]
Targeted behavior is likely to be amenable to feedback 6 (50) 0 (0) 6 (50) [52,53,55,60-62]
Self-efficacy 3 (25) 0 (0) 9 (75) [52,53,57]
Justify need for behavior change 10 (83) 0 (0) 2 (17) [52-55,58-62,64]
Rationale and added value
Rationale for using AFb 12 (100) 0 (0) 0 (0) [52-55,57-64]
Desirability, efficacy, safety, and cost-effectiveness 10 (83) 0 (0) 2 (17) [52,53,55,57-62,64]
Relative advantage 10 (83) 0 (0) 2 (17) [52,53,55,57-62,64]
Embedding in practice
Implementation complexity and compatibility with target behavior and work processes
Complexity of implementation process 8 (67) 1 (8) 3 (25) [54,58-64]
Technology supply model 8 (67) 0 (0) 4 (33) [53,54,58,60-64]
Compatibility 11 (92) 1 (8) 0 (0) [52,54,55,57-64]
Remove barriers 11 (92) 0 (0) 1 (8) [52,54,55,57-64]
Opportunity costs (including additional efforts to use technology) 3 (25) 1 (8) 8 (67) [58,60,62]
Available resources 6 (50) 2 (17) 4 (33) [52,57,58,61-63]
Inner and outer setting
Structural characteristics 1 (8) 0 (0) 11 (92) [62]
Networks and communications 2 (17) 0 (0) 10 (83) [61,62]
Culture 3 (25) 3 (25) 6 (50) [59,61,63]
Patient needs and resources 1 (8) 1 (8) 10 (83) [54]
Implementation
Planning 6 (50) 0 (0) 6 (50) [54,58-60,62,64]
Execution 5 (42) 0 (0) 7 (58) [54,58,60,62,64]
Formative evaluation
APM-AF system use
Intended use 1 (8) 1 (8) 10 (83) [64]
Actual use 3 (25) 2 (17) 7 (58) [58,60,64]
Development process and formative evaluations 11 (92) 1 (8) 0 (0) [52-55,57-60,62-64]
Harms or unintended effects 4 (33) 0 (0) 8 (67) [54,61,63,64]
Trialability 9 (75) 1 (8) 2 (17) [52-55,57,59,60,62,64]
Revisions and updating 6 (50) 1 (8) 5 (42) [52-54,57,60,64]
Replicability and digital preservation 1 (8) 1 (8) 10 (83) [64]

aAPM-AF: audit and feedback for antimicrobial resistance prevention measures.

bAF: audit and feedback.

 Stakeholders and Roles Research Team

Approximately 42% (5/12) of the studies described their research team, comprising multidisciplinary stakeholders (eg, HCWs, AMR experts, biostatisticians, and researchers) [53,55,57,59,61]. The research team compositions were substantiated (eg, having a multidisciplinary mix in the team [53,55,57,59,61] and experience with specific research methods [53,55,61]).

 Pilot Testers and Involvement in Development and Implementation Process

Pilot testers were described by 92% (11/12) of the studies [52-55,57,59-64] and were predominantly selected for their occupational function [52,55,57,59,60,62-64], whereas other details about stakeholders’ expertise and background were hardly described [52,53,57]. Stakeholder involvement was realized by including stakeholders (eg, HCWs and AMR experts) in the research team [62,63] and by involving them in pilot tests and formative evaluations [57,59,60,62-64]. Leadership engagement was mentioned by half of the studies as facilitator for successful implementation [54,58,61-63], whereas stakeholder engagement through champions or opinion leaders was mentioned less often (4/12, 33%) [52,58,61,62].

 Target Behavior and Added Value Target Behavior, Problem, and Intervention

The nature of the problem and relevant sociocultural factors were addressed by all studies [52-55,57-64]. Most studies (8/12, 67%) provided a description of underlying behavior and decision processes, either shortly in the article’s introduction [53,57] or in a prior study [52,55,58,59,62,63]. Some studies paid attention to whether there was tension for behavior change [52,53,55,60], whether the targeted behavior is likely to be amenable to feedback [52,53,55,60-62], and self-efficacy of feedback recipients’ (ie, feeling capable and responsible for behavior improvement) [52,53,57]. The need for behavior change was justified by pointing out flaws in current behaviors [52-55,58-62,64].

 Rationale and Added Value

All studies described the rationale and added value of the APM-AF [52-55,57-64]. Approximately 58% (7/12) of the studies described recommendations from health authorities (eg, World Health Organization) or AF as a widely used intervention in health care in general as reasons for developing and implementing an APM-AF system [53-55,57,59,61,63]. Other studies (5/12, 42%) explained how APM-AF could solve problems and inefficiencies in the current situation [52,58,60,62,64]. The added value was described both in terms of expectations (eg, improving the efficiency of audits) and experiences (eg, feedback motivated to change behavior) [52,53,55,57-62,64].

 Embedding in Practice Implementation Complexity and Compatibility With Target Behavior and Work Processes

Most studies (8/12, 67%) described constructs related to implementation complexity, including required organizational configurations and dependability on suppliers for customizations in terms of expected or experienced implementation barriers [53,54,58-64]. One of the studies specifically reflected on the duration and effort of the entire implementation process [58]. Approximately all studies (11/12, 92%) described constructs regarding the compatibility between the APM-AF system and stakeholders’ needs and existing workflows and expected or experienced implementation facilitators and barriers [52,54,55,57-64]. Opportunity costs were described by a few studies (3/12, 25%) [58,60,62], including negative experiences with the required additional efforts to use the APM-AF (including education and collecting data) [60,62]. A lack of resources was described in terms of staffing, time, and materials [57,58,61-63].

 Inner and Outer Setting

Few studies (4/12, 33%) paid attention to the inner setting, expressing the need for a collaborative environment and open culture, in which the quality of their work can be discussed safely [59,61,63]. One of the studies described increased patient involvement as a result of visible APM-AF systems [54].

 Implementation Planning and Execution

Approximately 50% (6/12) of the studies addressed implementation planning, of which 83% (5/6) also reflected on execution [54,58-60,62,64]. Studies mostly reported longer implementation processes than initially planned because of hospital workflow conflicts, personnel availability, and other confounding factors (including technical problems and resistance from stakeholders).

 Formative Evaluation APM-AF System Use

Intended and actual use of the APM-AF system was hardly (3/12, 25%) described, either as the maximum time HCWs should spend on filling out the audit tool [64] or as how often and complete the audit tool was filled in [58]. In addition, one of the studies used Google Analytics to measure users’ interactions with gamification parts [60].

 Development Process and Formative Evaluations

The development process and methods used for the formative evaluations were described in all studies [52-55,57-64] and elaborated in this paper’s Study Characteristics section.

 Harms or Unintended Effects

Approximately 33% (4/12) of the studies described whether and how harms and unintended effects were monitored during the development and implementation process (in general terms or with specific examples) [54,61,63,64].

 Trialability and Revisions and Updating

Approximately 75% (9/12) of the studies described trialability and revisions and updates in terms of several testing rounds [52-55,57,59,60,62,64]. However, only 50% (6/12) of the studies subsequently described, either specifically [57,60,64] or broadly [52-54], how the findings from the testing rounds were incorporated in the design or implementation of the APM-AF system (eg, use of better beacons).

 Replicability and Digital Preservation

One of the studies published their APM-AF system on the web with additional information (eg, web forum) [64].

 Discussion Principal Findings

This scoping review aimed to provide insights into strategies and theoretical underpinnings for AF system development and implementation from a sociotechnical perspective. Of the 2125 studies found in the search, 12 (0.56%) studies were included describing the development and implementation of their AF systems heterogeneously in terms of study aims, AF targets, and development and implementation strategies. Approximately 17% (2/12) of the studies explicitly aimed to illustrate how TMFs could guide choices in AF system development and implementation. A comprehensive interdisciplinary conceptual framework, based on overlapping and complementary constructs from TMFs from AF, eHealth and interventions, and implementation literature, was created to compare the studies.

 Lessons Learned for the Development and Implementation of APM-AF Systems (Research Question 5)

In this discussion, research question 5 is answered by providing lessons learned from reflecting upon our findings for theoretical underpinnings of and reporting on AF, AF systems, and their development and implementation.

Theoretical Underpinnings for AF

With health- and health care–related problems becoming increasingly complex, interdisciplinary theoretical integration to combine different perspectives is inevitable and pivotal to grasp the complexity of real-world problems [65]. This study showed the added value of considering and combining AF, eHealth, and implementation literature for studying AF systems. AF literature covered mostly AF system constructs (21/29, 72%), whereas relevant development and implementation constructs were hardly covered (12/35, 34%). Therefore, studies using only AF literature might miss important development and implementation constructs, such as stakeholder roles (eg, leadership engagement and champions) and embedding in practice (eg, implementation complexity and setting), that influence AF effectivity and efficiency [66,67]. Therefore, we argue that TMFs for AF and for development and implementation should be balanced, as exemplified in our interdisciplinary conceptual framework, and matched with studies’ specific research objectives, methods, and context (eg, setting and participants) [68].

However, selecting and combining the best-fitting TMFs remains a challenge [69]. Well-known examples of combined frameworks exist in implementation science (eg, Theoretical Domains Framework [70]); however, little information is provided about how constructs were combined. Overall, there is little guidance on the selection and integration of interdisciplinary TMFs [71]. Evolving initiatives that create shared languages across disciplines and theories (eg, CohenMiller and Pate [72]) and provide criteria for theory selection (eg, Birken et al [73]) are encouraged.

 Reporting on AF Systems

This scoping review resulted in an overview of constructs for APM-AF systems (Table 4), enriching the AF best practices from Colquhoun et al [18] with constructs of audit (eg, auditees and audit input), feedback framing and incentives, and AF system constructs (eg, technology and materials, human–system interaction, and data validity and trust and credibility). For replicability and using the framework in actual development and implementation, reporting about the audit input and what technology and materials were used is important. Furthermore, data validity and credibility was deemed as one of the 5 most important aspects for AF in a recent study [42].

In our view, the 2 constructs that were underreported in the included studies require attention in future studies. First, we observed quite broad descriptions of AF system design, with a lack of attention to functional (ie, what can the AF system do) and visual (ie, how efficiently and effectively information is presented to users) design, and engagement with the AF system. However, these constructs are important for how an AF system is received and used in practice [74]. The lack of design aspects and considerations of engagement might reflect the neglect of eHealth system characteristics (such as design) and engagement as active influencers for eHealth effectivity [75]. Second, comparison, goal setting, and action planning were hardly described in the included studies. A lack of substantiations for comparisons was also reported by a review on clinical performance comparators for AF on various clinical topics [76], whereas the underuse of goal setting and action planning was also found by a systematic review on behavior change interventions for APM in hospitals [28]. These 3 constructs were derived from all included theoretical perspectives and are common behavior change techniques [77], suggesting that they require and deserve more attention in future studies.

 Reporting on Development and Implementation of AF Systems

Of the full-text studies assessed for eligibility, most studies (30/58, 52%) were excluded as they focused primarily on effectiveness and did not sufficiently report on development and implementation. The inclusion of 12 studies in full-text might seem little; however, we believe this is exemplary of the tendency in (APM and eHealth) research to publish more about effectiveness than about the development and implementation process [3,11,12]. Therefore, in future AF system studies, but also for other eHealth technologies, considering development and implementation as influencers of the effectivity and efficiency of eHealth in practice has yet to gain ground [13]. During the construction of the conceptual framework, the interwovenness of development, implementation, and formative evaluation became apparent through the many overlaps observed. This resonates with best practices in eHealth development and implementation, which state that implementation is integrated with development and requires an iterative and holistic approach [78]. Therefore, next to reporting on evaluation, studies should report on both the constructs for AF systems and for development and implementation.

There is no single right development and implementation approach because of the many variations in APM and AF objectives, stakeholders, technologies, and settings [79]. Therefore, the constructed conceptual framework should be seen as a checklist that provides general guidance for potentially interesting constructs to consider; it remains at the discretion of researchers and developers which and how constructs are incorporated in their study. At a minimum, we propose to reflect upon relevant stakeholders and their roles, implementation complexity, compatibility with target behavior, and work processes, including the added value of AF and formative evaluation. Overall, supporting researchers and HCWs in selecting and integrating TMFs into their development and implementation process and promoting the explicit reporting of the theoretical underpinning and substantiation of choices are highly encouraged [41].

The constructed conceptual framework can be used in future studies to ensure a comprehensive view of AF for the transparency and replicability of individual studies. Therefore, we recommend using the conceptual framework as a checklist and adding it (including substantiation of choices) as a supplementary material in future publications. Furthermore, findings from this study can be used to improve the professionalization and harmonization of AF studies, which is important considering the increasing use of AF principles in learning health care systems [80]. The lack of attention to factors that support the development of learning health care systems (eg, organizational culture and cooperation between HCWs and researchers) is recognized as an important barrier to the widespread adoption of learning health care systems [81]. As these aspects are included in the constructed conceptual framework, it might help future researchers and developers to explicitly consider and integrate these constructs in their AF or learning system.

 Limitations and Strengths

This scoping review has several limitations. Although a comprehensive search query was used to search the most important databases for health research, we only included peer-reviewed published research and excluded evaluation studies. As a result, it might be possible that relevant findings were omitted (eg, from gray literature). Two systematic reviews on AF for various health targets were screened to ensure that no relevant publications were missed [1,2]. Another limitation is that evaluation studies were excluded from this review to highlight constructs relevant to the development and implementation, whereas evaluation is critical to know whether an intervention was successful. Therefore, it will be an important next step to evaluate AF systems in terms of processes (eg, improved HH), clinical outcomes (eg, reduced number of infections and decreased AMR), and technological outcomes (eg, APM-AF system use and persuasiveness) [82]. Finally, data extraction relied on the subjective interpretation of the constructs from the included publications by 1 researcher. However, the conceptual framework (Table 2 and Table 3) provided a thorough base for systematic and structured assessment, and the findings were iteratively discussed and revised throughout the review process.

 Conclusions

This scoping review provides novel insights into the theoretical underpinning of and reporting on the development and implementation of AF systems while demonstrating how a comprehensive conceptual framework can be created and used and a valuable means for capturing relevant constructs from heterogeneous studies with varying theoretical underpinnings. Few studies have explicitly described how choices for AF systems and their development and implementation processes were substantiated by TMFs. The interdisciplinary conceptual framework developed in this study is a first step toward the professionalization and harmonization of AF development and implementation with a sociotechnical approach. It provides guidance and a comprehensive checklist to guide researchers, HCWs, and policy makers in making informed choices in the development and implementation of AF systems, with the aim of further improving the quality and safety of health care.

Eligibility criteria.

Data extraction form.

Audit and feedback for antimicrobial resistance prevention measures systems.

Audit and feedback for antimicrobial resistance prevention measures development and implementation.

 Abbreviations AF

audit and feedback

 AMR

antimicrobial resistance

 APM

antimicrobial resistance prevention measures

 APM-AF

audit and feedback for antimicrobial resistance prevention measures

 HCW

health care worker

 HH

hand hygiene

 PRISMA-ScR

Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews

 TMF

theory, model, and framework

This research was supported by the INTERREG-VA (grant 202085) funded project EurHealth-1Health, which is part of a Dutch–German cross-border network supported by the European Commission; the Dutch Ministry of Health, Welfare and Sport; the Ministry of Economy, Innovation, Digitalization and Energy of the German Federal State of North Rhine-Westphalia; and the Ministry for National and European Affairs and Regional Development of Lower Saxony.

JK and NBJ conceptualized the manuscript. JK, BEB, and NBJ performed the data analysis. JK, NBJ, BEB, and LJEWCGP discussed the assessment form and results. NBJ, NAN, and LJEWCGP supervised the work. JK wrote the original draft, and all authors reviewed and edited the final manuscript.

None declared.

 Ivers N Jamtvedt G Flottorp S Young JM Odgaard-Jensen J French SD O'Brien MA Johansen M Grimshaw J Oxman AD Audit and feedback: effects on professional practice and healthcare outcomes Cochrane Database Syst Rev 2012 06 13 6 CD000259 10.1002/14651858.CD000259.pub3 22696318 Tuti T Nzinga J Njoroge M Brown B Peek N English M Paton C van der Veer SN A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory Implement Sci 2017 05 12 12 1 61 10.1186/s13012-017-0590-z 28494799 10.1186/s13012-017-0590-z PMC5427645 Ivers NM Sales A Colquhoun H Michie S Foy R Francis JJ Grimshaw JM No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention Implement Sci 2014 01 17 9 14 10.1186/1748-5908-9-14 24438584 1748-5908-9-14 PMC3896824 Colquhoun HL Carroll K Eva KW Grimshaw JM Ivers N Michie S Sales A Brehaut JC Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses Implement Sci 2017 09 29 12 1 117 10.1186/s13012-017-0646-0 28962632 10.1186/s13012-017-0646-0 PMC5622490 Hysong SJ Best RG Pugh JA Audit and feedback and clinical practice guideline adherence: making feedback actionable Implement Sci 2006 04 28 1 9 10.1186/1748-5908-1-9 16722539 1748-5908-1-9 PMC1479835 Kluger AN DeNisi A The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory Psychol Bulletin 1996 03 119 2 254 84 10.1037/0033-2909.119.2.254 Davey P Marwick CA Scott CL Charani E McNeil K Brown E Gould IM Ramsay CR Michie S Interventions to improve antibiotic prescribing practices for hospital inpatients Cochrane Database Syst Rev 2017 02 09 2 CD003543 10.1002/14651858.CD003543.pub4 28178770 PMC6464541 Neo JR Niederdeppe J Vielemeyer O Lau B Demetres M Sadatsafavi H Evidence-based strategies in using persuasive interventions to optimize antimicrobial use in healthcare: a narrative review J Med Syst 2020 02 10 44 3 64 10.1007/s10916-020-1531-y 32040649 10.1007/s10916-020-1531-y Livorsi DJ Heintz B Jacob JT Krein SL Morgan DJ Perencevich EN Audit and feedback processes among antimicrobial stewardship programs: a survey of the society for healthcare epidemiology of America research network Infect Control Hosp Epidemiol 2016 06 10 37 6 704 6 10.1017/ice.2016.57 26961763 S0899823X1600057X Baysari MT Lehnbom EC Li L Hargreaves A Day RO Westbrook JI The effectiveness of information technology to improve antimicrobial prescribing in hospitals: a systematic review and meta-analysis Int J Med Inform 2016 08 92 15 34 10.1016/j.ijmedinf.2016.04.008 27318068 S1386-5056(16)30065-X van Limburg M van Gemert-Pijnen JEWC Nijland N Ossebaard HC Hendrix RM Seydel ER Why business modeling is crucial in the development of eHealth technologies J Med Internet Res 2011 12 28 13 4 e124 10.2196/jmir.1674 22204896 v13i4e124 PMC3278110 Kip H Kelders SM Bouman YH van Gemert-Pijnen JEWC The importance of systematically reporting and reflecting on eHealth development: participatory development process of a virtual reality application for forensic mental health care J Med Internet Res 2019 08 19 21 8 e12972 10.2196/12972 31429415 v21i8e12972 PMC6718085 van Gemert-Pijnen JEWC Kelders SM Kip H Sanderman R eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge van Gemert-Pijnen JEWC Nijland N van Limburg M Ossebaard HC Kelders SM Eysenbach G Seydel ER A holistic framework to improve the uptake and impact of eHealth technologies J Med Internet Res 2011 12 05 13 4 e111 10.2196/jmir.1672 22155738 v13i4e111 PMC3278097 Keizer J Beerlage-De Jong N Al Naiemi N van Gemert-Pijnen JEWC Finding the match between healthcare worker and expert for optimal audit and feedback on antimicrobial resistance prevention measures Antimicrob Resist Infect Control 2020 08 05 9 1 125 10.1186/s13756-020-00794-7 32758300 10.1186/s13756-020-00794-7 PMC7405438 Core elements of hospital antibiotic stewardship programs Centers for Disease Control and Prevention 2019 2022-03-01 https://www.cdc.gov/antibiotic-use/core-elements/hospital.html Glanz K Bishop DB The role of behavioral science theory in development and implementation of public health interventions Annu Rev Public Health 2010 31 399 418 10.1146/annurev.publhealth.012809.103604 20070207 Colquhoun H Michie S Sales A Ivers N Grimshaw JM Carroll K Chalifoux M Eva K Brehaut J Reporting and design elements of audit and feedback interventions: a secondary review BMJ Qual Saf 2017 01 25 26 1 54 60 10.1136/bmjqs-2015-005004 26811541 bmjqs-2015-005004 Gardner B Whittington C McAteer J Eccles MP Michie S Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback Soc Sci Med 2010 05 70 10 1618 25 10.1016/j.socscimed.2010.01.039 20207464 S0277-9536(10)00122-X Webb TL Joseph J Yardley L Michie S Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy J Med Internet Res 2010 02 17 12 1 e4 10.2196/jmir.1376 20164043 v12i1e4 PMC2836773 Morrison LG Theory-based strategies for enhancing the impact and usage of digital health behaviour change interventions: a review Digit Health 2015 1 2055207615595335 10.1177/2055207615595335 29942544 10.1177_2055207615595335 PMC5999061 Global Action Plan on Antimicrobial Resistance 2015 Geneva World Health Organization O'Neill J Antimicrobial Resistance: Tackling A Crisis for the Health and Wealth of Nations 2014 London Review on Antimicrobial Resistance Dik JH Poelman R Friedrich AW Panday PN Lo-Ten-Foe JR van Assen S van Gemert-Pijnen JEWC Niesters HG Hendrix R Sinha B An integrated stewardship model: antimicrobial, infection prevention and diagnostic (AID) Future Microbiol 2016 11 1 93 102 10.2217/fmb.15.99 26323589 McCullough AR Rathbone J Parekh S Hoffmann TC Del Mar CB Not in my backyard: a systematic review of clinicians' knowledge and beliefs about antibiotic resistance J Antimicrob Chemother 2015 09 70 9 2465 73 10.1093/jac/dkv164 26093375 dkv164 Keizer J Braakman-Jansen LM Kampmeier S Köck R Al Naiemi N Te Riet-Warning R Beerlage-De Jong N Becker K Van Gemert-Pijnen JEWC Cross-border comparison of antimicrobial resistance (AMR) and AMR prevention measures: the healthcare workers’ perspective Antimicrob Resist Infect Control 2019 7 22 8 1 123 10.1186/s13756-019-0577-4 Colquhoun HL Brehaut JC Sales A Ivers N Grimshaw J Michie S Carroll K Chalifoux M Eva KW A systematic review of the use of theory in randomized controlled trials of audit and feedback Implement Sci 2013 06 10 8 66 10.1186/1748-5908-8-66 23759034 1748-5908-8-66 PMC3702512 Davey P Peden C Charani E Marwick C Michie S Time for action-Improving the design and reporting of behaviour change interventions for antimicrobial stewardship in hospitals: early findings from a systematic review Int J Antimicrob Agents 2015 03 45 3 203 12 10.1016/j.ijantimicag.2014.11.014 25630430 S0924-8579(14)00401-4 Lorencatto F Charani E Sevdalis N Tarrant C Davey P Driving sustainable change in antimicrobial prescribing practice: how can social and behavioural sciences help? J Antimicrob Chemother 2018 10 01 73 10 2613 24 10.1093/jac/dky222 30020464 5055403 Edwards R Charani E Sevdalis N Alexandrou B Sibley E Mullett D Loveday HP Drumright LN Holmes A Optimisation of infection prevention and control in acute health care by use of behaviour change: a systematic review Lancet Infectious Diseases 2012 04 12 4 318 29 10.1016/s1473-3099(11)70283-3 Hulscher ME Grol RP van der Meer JW Antibiotic prescribing in hospitals: a social and behavioural scientific approach Lancet Infectious Diseases 2010 03 10 3 167 75 10.1016/s1473-3099(10)70027-x Cruz-Martínez RR Wentzel J Asbjørnsen RA Noort PD van Niekerk JM Sanderman R van Gemert-Pijnen JEWC Supporting self-management of cardiovascular diseases through remote monitoring technologies: metaethnography review of frameworks, models, and theories used in research and development J Med Internet Res 2020 05 21 22 5 e16157 10.2196/16157 32436852 v22i5e16157 PMC7273239 van Buul LW Sikkens JJ van Agtmael MA Kramer MH van der Steen JT Hertogh CM Participatory action research in antimicrobial stewardship: a novel approach to improving antimicrobial prescribing in hospitals and long-term care facilities J Antimicrob Chemother 2014 07 69 7 1734 41 10.1093/jac/dku068 24648505 dku068 Gregor S Hevner AR Positioning and presenting design science research for maximum impact MISQ 2013 2 2 37 2 337 55 10.25300/misq/2013/37.2.01 Beerlage-de Jong N Eikelenboom-Boskamp A Voss A Sanderman R van Gemert-Pijnen JEWC Combining user-centered design with the persuasive systems design model: The development process of a web-based registration and monitoring system for healthcare-associated infections in nursing homes Int J Advances Life Sci 2018 6 3 262 71 Oinas-Kukkonen H Harjumaa M Persuasive systems design: key issues, process model, and system features Commun Assoc Inf Syst 2009 24 10.17705/1cais.02428 Wentzel J van Drie-Pierik R Nijdam L Geesing J Sanderman R van Gemert-Pijnen JEWC Antibiotic information application offers nurses quick support Am J Infect Control 2016 06 01 44 6 677 84 10.1016/j.ajic.2015.12.038 26905792 S0196-6553(16)00016-X Leis JA Advancing infection prevention and antimicrobial stewardship through improvement science BMJ Qual Saf 2018 02 14 27 2 163 5 10.1136/bmjqs-2017-006793 28615344 bmjqs-2017-006793 Birgand G Johansson A Szilagyi E Lucet J Overcoming the obstacles of implementing infection prevention and control guidelines Clin Microbiol Infect 2015 12 21 12 1067 71 10.1016/j.cmi.2015.09.005 26369604 S1198-743X(15)00820-4 Storr J Twyman A Zingg W Damani N Kilpatrick C Reilly J Price L Egger M Grayson ML Kelley E Allegranzi B WHO Guidelines Development Group Core components for effective infection prevention and control programmes: new WHO evidence-based recommendations Antimicrob Resist Infect Control 2017 6 6 10.1186/s13756-016-0149-9 28078082 149 PMC5223492 Davies P Walker AE Grimshaw JM A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations Implement Sci 2010 02 09 5 14 10.1186/1748-5908-5-14 20181130 1748-5908-5-14 PMC2832624 Colquhoun HL Carroll K Eva KW Grimshaw JM Ivers N Michie S Brehaut JC Informing the research agenda for optimizing audit and feedback interventions: results of a prioritization exercise BMC Med Res Methodol 2021 01 13 21 1 20 10.1186/s12874-020-01195-5 33435873 10.1186/s12874-020-01195-5 PMC7805176 Eysenbach G CONSORT-EHEALTH Group CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions J Med Internet Res 2011 12 31 13 4 e126 10.2196/jmir.1923 22209829 v13i4e126 PMC3278112 Arksey H O'Malley L Scoping studies: towards a methodological framework Int J Soc Res Methodol 2005 02 8 1 19 32 10.1080/1364557032000119616 Peters MD Godfrey CM Khalil H McInerney P Parker D Soares CB Guidance for conducting systematic scoping reviews Int J Evid Based Healthc 2015 09 13 3 141 6 10.1097/XEB.0000000000000050 26134548 Tricco AC Lillie E Zarin W O'Brien KK Colquhoun H Levac D Moher D Peters MD Horsley T Weeks L Hempel S Akl EA Chang C McGowan J Stewart L Hartling L Aldcroft A Wilson MG Garritty C Lewin S Godfrey CM Macdonald MT Langlois EV Soares-Weiser K Moriarty J Clifford T Tunçalp ? Straus SE PRISMA extension for Scoping Reviews (PRISMA-ScR): checklist and explanation Ann Intern Med 2018 09 04 169 7 467 10.7326/M18-0850 Eysenbach G What is e-health? J Med Internet Res 2001 3 2 E20 10.2196/jmir.3.2.e20 11720962 PMC1761894 Hoffmann TC Glasziou PP Boutron I Milne R Perera R Moher D Altman DG Barbour V Macdonald H Johnston M Lamb SE Dixon-Woods M McCulloch P Wyatt JC Chan A Michie S Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide BMJ 2014 03 07 348 g1687 10.1136/bmj.g1687 24609605 Damschroder LJ Aron DC Keith RE Kirsh SR Alexander JA Lowery JC Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science Implement Sci 2009 08 07 4 50 10.1186/1748-5908-4-50 19664226 1748-5908-4-50 PMC2736161 Greenhalgh T Wherton J Papoutsi C Lynch J Hughes G A'Court C Hinder S Fahy N Procter R Shaw S Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies J Med Internet Res 2017 11 01 19 11 e367 10.2196/jmir.8775 29092808 v19i11e367 PMC5688245 Tong A Flemming K McInnes E Oliver S Craig J Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ BMC Med Res Methodol 2012 11 27 12 1 181 10.1186/1471-2288-12-181 23185978 1471-2288-12-181 PMC3552766 Boscart VM Fernie GR Lee JH Jaglal SB Using psychological theory to inform methods to optimize the implementation of a hand hygiene intervention Implement Sci 2012 08 28 7 1 77 10.1186/1748-5908-7-77 22929925 1748-5908-7-77 PMC3503739 Conway LJ Riley L Saiman L Cohen B Alper P Larson EL Implementation and impact of an automated group monitoring and feedback system to promote hand hygiene among health care personnel Joint Commission J Qual Patient Safety 2014 09 40 9 408 17 10.1016/s1553-7250(14)40053-9 Edmisten C Hall C Kernizan L Korwek K Preston A Rhoades E Shah S Spight L Stradi S Wellman S Zygadlo S Implementing an electronic hand hygiene monitoring system: lessons learned from community hospitals Am J Infect Control 2017 08 01 45 8 860 5 10.1016/j.ajic.2017.03.033 28526308 S0196-6553(17)30289-4 Hysong SJ Kell HJ Petersen LA Campbell BA Trautner BW Theory-based and evidence-based design of audit and feedback programmes: examples from two clinical intervention studies BMJ Qual Saf 2017 04 10 26 4 323 34 10.1136/bmjqs-2015-004796 27288054 bmjqs-2015-004796 Smither JW London M Richmond KR The relationship between leaders’ personality and their reactions to and use of multisource feedback Group Organization Manag 2016 07 26 30 2 181 210 10.1177/1059601103254912 James R Upjohn L Cotta M Luu S Marshall C Buising K Thursky K Measuring antimicrobial prescribing quality in Australian hospitals: development and evaluation of a national antimicrobial prescribing survey tool J Antimicrob Chemother 2015 02 26 70 6 1912 8 10.1093/jac/dkv047 25724986 dkv047 Jeanes A Coen PG Drey NS Gould DJ Moving beyond hand hygiene monitoring as a marker of infection prevention performance: development of a tailored infection control continuous quality improvement tool Am J Infect Control 2020 01 48 1 68 76 10.1016/j.ajic.2019.06.014 31358420 S0196-6553(19)30645-5 PMC7115327 Keizer J Beerlage-de Jong N Al Naiemi N van Gemert-Pijnen JEWC Persuading from the start: participatory development of sustainable persuasive data-driven technologies in healthcare Persuasive Technology. Designing for Future Change 2020 Cham Springer Marques R Gregório J Pinheiro F Póvoa P da Silva MM Lapão LV How can information systems provide support to nurses' hand hygiene performance? Using gamification and indoor location to improve hand hygiene awareness and reduce hospital infections BMC Med Inform Decis Mak 2017 01 31 17 1 15 10.1186/s12911-017-0410-z 28143613 10.1186/s12911-017-0410-z PMC5282776 Pakyz AL Moczygemba LR VanderWielen LM Edmond MB Stevens MP Kuzel AJ Facilitators and barriers to implementing antimicrobial stewardship strategies: results from a qualitative study Am J Infect Control 2014 10 42 10 Suppl S257 63 10.1016/j.ajic.2014.04.023 25239719 S0196-6553(14)00734-2 Parker V Giles M King J Bantawa K Barriers and facilitators to implementation of a multifaceted nurse-led intervention in acute care hospitals aimed at reducing indwelling urinary catheter use: a qualitative study J Clin Nurs 2020 08 14 29 15-16 3042 53 10.1111/jocn.15337 32441867 Patel SJ Saiman L Duchon JM Evans D Ferng Y Larson E Development of an antimicrobial stewardship intervention using a model of actionable feedback Interdiscip Perspect Infect Dis 2012 2012 150367 10.1155/2012/150367 22500166 PMC3303556 Power M Fogarty M Madsen J Fenton K Stewart K Brotherton A Cheema K Harrison A Provost L Learning from the design and development of the NHS Safety Thermometer Int J Qual Health Care 2014 06 30 26 3 287 97 10.1093/intqhc/mzu043 24787136 mzu043 PMC4041095 Nilsen P Making sense of implementation theories, models and frameworks Implement Sci 2015 04 21 10 53 10.1186/s13012-015-0242-0 25895742 10.1186/s13012-015-0242-0 PMC4406164 Pulcini C Binda F Lamkang AS Trett A Charani E Goff DA Harbarth S Hinrichsen SL Levy-Hara G Mendelson M Nathwani D Gunturu R Singh S Srinivasan A Thamlikitkul V Thursky K Vlieghe E Wertheim H Zeng M Gandra S Laxminarayan R Developing core elements and checklist items for global hospital antimicrobial stewardship programmes: a consensus approach Clin Microbiol Infect 2019 01 25 1 20 5 10.1016/j.cmi.2018.03.033 29625170 S1198-743X(18)30295-7 Guidelines on Core Components of Infection Prevention and Control Programmes at the National and Acute Health Care Facility Level 2017 Genève, Switzerland World Health Organization Kip H Keizer J da Silva MC Beerlage-de Jong N Köhle N Kelders S Methods for human-centered eHealth development: narrative scoping review J Med Internet Res 2022 01 27 24 1 e31858 10.2196/31858 35084359 v24i1e31858 Birken SA Powell BJ Presseau J Kirk MA Lorencatto F Gould NJ Shea CM Weiner BJ Francis JJ Yu Y Haines E Damschroder LJ Combined use of the Consolidated Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF): a systematic review Implement Sci 2017 01 05 12 1 2 10.1186/s13012-016-0534-z 28057049 10.1186/s13012-016-0534-z PMC5217749 Michie S Johnston M Abraham C Lawton R Parker D Walker A Making psychological theory useful for implementing evidence based practice: a consensus approach Qual Saf Health Care 2005 02 14 1 26 33 10.1136/qshc.2004.011155 15692000 14/1/26 PMC1743963 Tobi H Kampen JK Research design: the methodology for interdisciplinary research framework Qual Quant 2018 52 3 1209 25 10.1007/s11135-017-0513-8 29674791 513 PMC5897493 CohenMiller AS Pate E A model for developing interdisciplinary research theoretical frameworks Qual Report 2019 6 2 24 6 1211 10.46743/2160-3715/2019.3558 Birken SA Powell BJ Shea CM Haines ER Alexis Kirk M Leeman J Rohweder C Damschroder L Presseau J Criteria for selecting implementation science theories and frameworks: results from an international survey Implement Sci 2017 10 30 12 1 124 10.1186/s13012-017-0656-y 29084566 10.1186/s13012-017-0656-y PMC5663064 Yigitbasioglu O Velcu O A review of dashboards in performance management: implications for design and research Int J Account Inf Syst 2012 3 13 1 41 59 10.1016/j.accinf.2011.08.002 Kelders SM Kip H Greeff J Psychometric evaluation of the TWente Engagement with Ehealth Technologies Scale (TWEETS): evaluation study J Med Internet Res 2020 10 09 22 10 e17757 10.2196/17757 33021487 v22i10e17757 PMC7576538 Gude WT Brown B van der Veer SN Colquhoun HL Ivers NM Brehaut JC Landis-Lewis Z Armitage CJ de Keizer NF Peek N Clinical performance comparators in audit and feedback: a review of theory and evidence Implement Sci 2019 04 24 14 1 39 10.1186/s13012-019-0887-1 31014352 10.1186/s13012-019-0887-1 PMC6480497 Carey R Connell L Johnston M Rothman A de Bruin M Kelly M Michie S Behavior change techniques and their mechanisms of action: a synthesis of links described in published intervention literature Ann Behav Med 2019 07 17 53 8 693 707 10.1093/abm/kay078 30304386 5126198 PMC6636886 Kip H van Gemert-Pijnen JEWC Holistic development of eHealth technology eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge 131 166 Kip H Beerlage-de Jong N Wentzel J The contextual inquiry eHealth Research, Theory and Development: A Multi-Disciplinary Approach 2018 London Routledge 167 186 Realising the potential of learning health systems The Health Foundation 2021 2022-03-01 https://learninghealthcareproject.org/wp-content/uploads/2021/05/LHS2021report.pdf Mapping the Learning Health System: A Scoping Review of Current Evidence. A White Paper 2020 New South Wales, Australia Australian Institute of Health Innovation, Macquarie University Beerlage-de Jong N eHealth vs Infection: Participatory Development of Persuasive eHealth to Support Safe Care 2016 Enschede, The Netherlands University of Twente