Background

JMIR

J Med Internet Res

Journal of Medical Internet Research

1438-8871

JMIR Publications

Toronto, Canada

v27i1e62763

39918864

10.2196/62763

Review

Understanding “Alert Fatigue” in Primary Care: Qualitative Systematic Review of General Practitioners Attitudes and Experiences of Clinical Alerts, Prompts, and Reminders

Cahill

Naomi

Marshall

Robert

P?rez Jover

Virtudes

Gani

Illin

MPH 1

https://orcid.org/0009-0008-3644-278X

Litchfield

Ian

PhD 1

Department of Health Sciences University of Birmingham

Room 134

Murray Learning Centre

Birmingham, B15 2TT

United Kingdom 44 01214146006 i.litchfield@bham.ac.uk

https://orcid.org/0000-0002-1169-5392

Shukla

David

MBChB, MRCGP 1

https://orcid.org/0000-0001-6229-7477

Delanerolle

Gayathri

PhD 1

https://orcid.org/0000-0002-9628-9245

Cockburn

Neil

MBChB 1

https://orcid.org/0000-0001-9284-6991

Pathmanathan

Anna

MPH 2

https://orcid.org/0009-0001-7981-4132

1 Department of Health Sciences University of Birmingham

Birmingham

United Kingdom 2 Population Health Sciences Centre for Academic Primary Care University of Bristol

Bristol

United Kingdom

Corresponding Author: Ian Litchfield i.litchfield@bham.ac.uk

2025

7 2 2025

e62763

3 6 2024 27 9 2024 8 10 2024 23 12 2024

©Illin Gani, Ian Litchfield, David Shukla, Gayathri Delanerolle, Neil Cockburn, Anna Pathmanathan. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.02.2025.

2025

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Background

The consistency and quality of care in modern primary care are supported by various clinical reminders (CRs), which include “alerts” describing the consequences of certain decisions and “prompts” that remind users to perform tasks promoting desirable clinical behaviors. However, not all CRs are acted upon, and many are disregarded by general practitioners (GPs), a chronic issue commonly referred to as “alert fatigue.” This phenomenon has significant implications for the safety and quality of care, GP burnout, and broader medicolegal consequences. Research on mitigating alert fatigue and optimizing the use of CRs remains limited. This review offers much-needed insight into GP attitudes toward the deployment, design, and overall effectiveness of CRs.

Objective

This systematic review aims to synthesize current qualitative research on GPs’ attitudes toward CRs, enabling an exploration of the interacting influences on the occurrence of alert fatigue in GPs, including the deployment, design, and perceived efficacy of CRs.

Methods

A systematic literature search was conducted across the Health Technology Assessment database, MEDLINE, MEDLINE In-Process, Embase, CINAHL, Conference Proceedings Citation Index, PsycINFO, and OpenGrey. The search focused on primary qualitative and mixed methods research conducted in general or family practice, specifically exploring GPs’ experiences with CRs. All databases were searched from inception to December 31, 2023. To ensure structured and practicable findings, we used a directed content analysis of the data, guided by the 7 domains of the Non-adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework, including domains related to Technology, Adopter attitudes, and Organization.

Results

A total of 9 studies were included, and the findings were organized within the 7 domains. Regarding Condition and Value Proposition, GPs viewed CRs as an effective way to maintain or improve the safety and quality of care they provide. When considering the attributes of the Technology, the efficacy of CRs was linked to their frequency, presentation, and the accuracy of their content. Within Adopters, concerns were raised about the accuracy of CRs and the risk that their use could diminish the value of GP experience and contextual understanding. From an Organization perspective, the need for training on the use and benefits of CRs was highlighted. Finally, in the context of the Wider system and their Embedding Over Time, suggestions included sharing best practices for CR use and involving GPs in their design.

Conclusions

While GPs acknowledged that CRs, when used optimally, can enhance patient safety and quality of care, several concerns emerged regarding their design, content accuracy, and lack of contextual nuance. Suggestions to improve CR adherence included providing coherent training, enhancing their design, and incorporating more personalized content.

Trial Registration

PROSPERO CRD42016029418; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=29418

International Registered Report Identifier (IRRID)

RR2-10.1186/s13643-017-0627-z

primary care general practitioners alert fatigue computer decision support systems fatigue qualitative systematic review quality of care clinical behaviors behaviors database family practice family algorithm patient safety patient

Introduction

Primary care is facing unprecedented pressure across the globe, caught between growing patient demand, issues of workforce recruitment and retention, and increasingly complex options for treatment [1,2]. To improve the consistency and quality of care in this dynamic and highly pressured environment, health systems worldwide have introduced guidelines for care informed by the latest clinical evidence [3,4]. These guidelines are promoted and supported in high-income countries through a range of clinical reminders (CRs) that are presented to general practitioners (GPs) via their clinical management systems and consist of “alerts” that remind them of the consequences of making a certain decision, and “prompts” that remind the user to perform a task that promotes desirable clinical behaviors [5-10]. These alerts can relate to potential diagnoses, drug-drug interactions, and allergies, and prompts linked to patient-specific needs, further information, and recommendations for high-quality care (see Figure 1; see also [11]).

Figure 1

Types of clinical reminders and alerts used in clinical management systems in primary care (Cecil et al [11]).

The growing sophistication of medical knowledge and the electronic systems that manage patient data and inform clinical decision-making have led to a steady increase in the activation of CRs at the point of care and the cognitive load placed on GPs [12-15]. Not all CRs are acted upon, and some may justifiably be considered irrelevant or inapplicable [16,17], but increasingly GPs are disregarding pertinent or urgent CRs [18-20], a chronic negligence commonly referred to as “alert fatigue” [21].

Despite the implications of alert fatigue for the safety and quality of care, GP burnout, and the medico-legal consequences for individuals and their organizations, the proliferation of CRs in primary care is reliant on evidence limited to their effects on a specific condition [22,23]. What remains underexplored is how their use might be optimized in the context of the cumulative effect of multiple CRs on a single health care provider [19,20,22,24]. There is then a need to better understand GP attitudes toward the deployment, design, and overall effectiveness of CRs, as well as broader implications for their impact on patient-clinician communication [22,25]. The systematic review presented here aims to address this by synthesizing current qualitative research on GPs’ attitudes toward CRs describing the findings within the Non-adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework [26]. This framework was designed to study unfolding digital health technology programs and so provides the opportunity for offering structured insight into the future design, use, and positioning of CRs in primary care.

Methods Applying the NASSS Framework to Digital Health Implementation

The review used best practices in systematic review methodology and directed content analysis [27,28]. We describe the results through the lens of the NASSS framework developed to support, guide, and monitor the implementation of digital technologies in health and social care [26,29,30]. The framework uses 7 domains to describe various aspects of the technology, end users, organizational structure, and broader policy-driven context [26]. The domains are defined alongside their potential impact on implementation in Table 1.

Table 1

The NASSS^a framework: domains, definitions, and influences on implementation (adapted from [26,29-31]).

Domain	Definition	Influences on implementation
Condition	The condition(s) for which the innovation or technology has been designed. These can be physical, mental, or psychosocial in nature.	The complexity of the condition. Its metabolic volatility, association with comorbidities, and impact on cognitive function [32].
Technology	The technology/ies or other innovation that is/are being introduced. Include(s) both hardware and software and can include a novel protocol or pathway—or some combination of these.	Its material properties, functionality, dependability, and speed [33]. The knowledge needed to use them [34]. The knowledge generated by the technology [35]. The supply model and the relationship with the care provider [36]. The ownership of intellectual property [37].
Value Proposition	The value (financial or otherwise) that the new technology and care model generates. This includes, for commercial stakeholders, the return on investment; for patients, improvements in comfort or quality of life; for health care organizations, improvements in quality, safety, inclusivity, and efficiency of the care delivered.	Provision of value to a range of stakeholders, suppliers, patients, the health care system, and taxpayers or insurers. Formulating a credible business plan where efficacy or cost-effectiveness studies are unavailable or contested [38,39].
Adopters	The intended users of the technology or other innovation. This includes patients/laypersons, professionals, administrative, and support staff.	Acceptability to service users and their families/carers [40]. Attitudes toward new and emerging technologies [41]. Influence of sociocultural factors such as poverty or social exclusion [42]. Acceptability to staff. Impact on roles, professional traditions, and codes of conduct [26].
Organizations	The cultural and organizational characteristics of the organizations involved. This includes structure, capacity, and capability to adopt new ways of working, as well as resources of staff and infrastructure.	The organization’s general capacity to innovate [43]. Readiness for this particular technology [26]. The decisions around funding the intervention (including the presence of interorganizational agreements or speculative cross-system savings in the funding decision [44]). The extent of the change needed, including the potential disruption to existing routines [45]. The work required in implementation, including staff engagement, fidelity of implementation, and evaluation [46].
Wider System	The national and local context for the introduction of the technology or program	The impact of national and local policies and objectives [47]. The support of regulatory or professional bodies [48]. Sociocultural factors including public perceptions of the technology [49]. The presence of interorganizational networking and collaborative initiatives in supporting implementation [50].
Embedding Over Time	The key changes and uncertainties expected to affect the integration of the technology over the next 3-5 years.	The ability of the technology to adapt to changing contexts [51]. The resilience and cultural stability of the organizations involved [52].

^aNASSS: Non-adoption, Abandonment, Scale-up, Spread, and Sustainability.

Search Strategy

The databases searched were the Health Technology Assessment Database, MEDLINE, MEDLINE in Process, Embase, CINAHL, Conference Proceedings Citation Index, PsycINFO, and OpenGrey, from January 1, 1960, to December 31, 2023, a date range chosen to reflect the introduction of electronic or digitally supported health care. The search was conducted by the author IG in March 2024 and no language or location limits were applied. Medical Subject Headings (MeSH) terms and free-text words were used against the Sample, Phenomenon of Interest, Design, Evaluation, Research type framework [53] to collect relevant studies from the perspective of GPs in a primary care setting (see Multimedia Appendix 1). The search strategy is described in the PROSPERO registration and the previously published protocol [54]. The EndNote (Clarivate Plc) bibliographic software was used to organize all retrieved studies and remove any duplicates.

Eligibility Criteria

Studies included were primary qualitative research conducted in primary care settings and published in English. The research had to have a focus on general or family practice and be qualitative in nature because we were specifically interested in GPs’ experience with and perspectives of CRs (see Multimedia Appendix 2 for detailed eligibility criteria).

Study Selection and Data Extraction

We used the recommended full dual review approach, that is, 2 independent reviewers (IG and AP) screened the title, abstract, and then the full texts, as this has been shown to increase the number of eligible papers identified and more broadly minimize error in the selection process. The data were managed using an Excel spreadsheet (Microsoft Corporation), with any disagreements on the inclusion of a study consensually resolved through discussion with a third reviewer (IL). A data extraction form was designed to summarize study characteristics. The data extracted included information on study outcomes, study design, participants, and key findings.

Quality Appraisal

The quality and bias of included papers were determined by the Critical Appraisal Skills Programme checklist for qualitative studies and the Mixed Methods Appraisal Tool for mixed method studies [55]. Although the Mixed Methods Appraisal Tool cautions against the approach of labeling studies at low, medium, and high risk of bias, we used a scoring system to determine this labeling, for ease of interpretation between the 2 different tools. The quality appraisal was undertaken independently by 2 authors (IL and IG) who between them agreed on the final value.

Data Synthesis

The data were coded and managed via NVivo12 (Lumivero, LLC). We used a directed content analysis to populate the NASSS framework, based on the unconstrained matrix approach. This allowed the development and inclusion of constructs directly relevant to emergent themes within the established framework [56,57]. The analysis was conducted by IG and IL who both independently analyzed a sample of the same 3 papers. They then met to discuss their analysis and any discrepancies in the allocation of data before IL analyzed the remainder of the studies, with the final analysis and data allocation consensually agreed upon by all authors. In completing the analysis, we merged the domains of “Wider system” and “Embedding over time” due to the concurrence that arose in the data.

Results Study Characteristics

The screening and study selection are outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram (Figure 2; also see Multimedia Appendix 3). The study selection and data extraction templates can be found in Multimedia Appendix 4. Full-text studies were excluded because of duplication, or lack of detail on the care provider, setting, or digital tool (see Multimedia Appendix 5 for a detailed description). A total of 9 studies were included, 5 from the United Kingdom [58-62] and 1 each from Australia [63], Belgium [64], the Netherlands [65], and Norway [66]. Seven of these were considered high quality [58-62,65,66], 1 medium [64], and 1 low [63]. The papers were published between 2003 and 2021. The key characteristics of included studies and findings are summarized in Table 2. This includes the quality assessment (see Multimedia Appendix 6 for a detailed description). The directed content analysis enabled the mapping of the data within the NASSS framework. Within each domain, we determined a series of emergent constructs relevant to the implementation of CRs in primary care. These findings are summarized in Table 3 and presented below within each domain of the framework alongside exemplar quotes taken from the studies identified.

Figure 2

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram for the systematic review.

Table 2

Key characteristics and findings of identified studies.

Reference	Country of study	Study outcomes	Study design (quality of study)	Purpose of CR^a,b	Participants/setting	Key findings
Ahearn and Kerr [63]	Australia	To determine GPs’^c perceptions of strengths and weaknesses of prescription systems	Focus groups (low)	Medication	14 GPs; 1 focus group in a rural area and 2 focus groups in an urban area	CRs affected prescribing behavior. GPs recognized the need for training. Considered helpful features of clinical decision support systems.
Bindels et al [65]	The Netherlands	To investigate the experiences of GPs with the “GRIF system” (automated test ordering combined with a feedback system)	Group discussion and interview (high)	Diagnosis	24 GPs; mean age 49 (SD 4.9) years, mean experience 19 (SD 7) years; males comprised 75%	The system increased attention to and promotion of guidelines.
Christensen and Grimsmo [66]	Norway	To study primary care physicians’ experiences of electronic prescription reminders	Focus groups; observations of doctor-patient encounters; and National Questionnaire Survey (high)	Medication	24 GPs (15 females + 9 males)	The ability to present relevant patient information and medical knowledge was appreciated.
Dikomitis et al [61]	United Kingdom	To obtain views from GPs who piloted the electronic risk assessment tools	Telephone interviews (high)	Diagnosis	23 GPs (13 males + 10 females)	Key issues over the integration of CRs into existing digital platforms.
Ford et al [59]	United Kingdom	To understand the features of individual clinical decision support systems that were barriers and facilitators to their use	Semistructured interview (high)	Medication	11 GPs (7 males + 4 females); age group 30-39 years, n=8; age group 40-49 years, n=2; and age group 50+ years, n=1	Issues emerged relating to the accuracy of CRs and their usability.
Jeffries et al [58]	United Kingdom	To understand the factors that influenced clinical decision support system implementation	Semistructured interviews (high)	Medication	14 GPs across 4 regions in the United Kingdom	There was a shared recognition of the need to use CRs and their potential to support patient care but felt CRs needed to be more relevant.
Heselmans et al [64]	Belgium	To assess users’ perceptions toward the EBMeDS system (a computer decision support system provided by EBPNet, a national computerized point-of-care information service in Belgium)	Survey (medium)	Management, medication, and diagnosis	Survey issued to 334 (Dutch-speaking) GPs	Reminders considered too general and inflexible, with questions over their reliability.
Holt et al [60]	United Kingdom	To investigate the implementation of the CRs in UK general practice	Semistructured interview (high)	Diagnosis	7 GPs (no characteristics described from the sample)	The advantages of CRs were recognized but with caveats over their specificity.
Slight et al [62]	United Kingdom	To examine the causes of prescribing and monitoring errors in general practice	Semistructured interviews and focus groups (high)	Medication	33 GPs participated in semistructured interviews and 46 contributed in focus groups	The need for prior training and improved information technology infrastructure and hardware.

^aCR: clinical reminder.

^bMedication, diagnosis, or ongoing management.

^cGP: general practitioner.

Table 3

Summary of findings by NASSS^a domain.

NASSS domain and emergent constructs		Definition	Summary of findings (and review papers)
Condition
	Medication	To support safe prescribing and medication adherence.	Issues accommodated by CRs^b included drug-drug interactions, drug allergies, and dose-related recommendations [58,59,62-64,66].
	Diagnosis	To support prompt and accurate diagnosis of a range of chronic and acute conditions.	Diagnostic use included both acute [60,61] and chronic conditions [64,65].
	Ongoing management	Providing reminders and alerts for the safe and successful long-term management of (chronically ill) patients.	CRs included references to reviews of patient medication, symptoms, and testing regimes [64].
Technology
	Frequency and content of CRs	The number of CRs and the relevance and accuracy of their content.	The overabundance of CRs was described [59,61,63,64]. GPs^c described a preference for accompanying information that aided comprehension [59].
	Design and integration of CRs	The presentation and readability of CRs and their integration with existing clinical dashboards.	GPs described how CRs requiring a response disrupted their workflow [59,63]. A preference was expressed for CRs with clear color-coded designations of risk or otherwise readily understandable formatting [59]. Dropdown boxes were seen as overly complicated [62].
	Digital infrastructure	The capacity, capability, and compatibility of hardware and software systems and the interoperability of data.	The quality of connectivity and the capacity of practice software systems impacted the timely appearance of alerts, frustrating clinicians and increasing the risk of errors [59,60,62,66].
Value proposition
	Overall quality and safety	CRs capable of presenting timely prompts with the necessary actions.	CRs were valued for their ability to provoke a decision [58,61,63], particularly in busy environments [59,60]. Their ability to act as educational tools was mooted [58]. The constant interruption of tasks by CRs was considered a risk to patient safety [59].
	Self-efficacy	CRs can provide reassurance for clinicians that they have not missed anything significant.	The increasing complexity of primary care meant that clinicians welcomed the additional support [59,61,63].
Adopter
	Engagement, and compliance in CRs	The degree to which clinicians follow the recommendations of CRs and the factors that influence this decision.	There were concerns over the accuracy and relevance of CRs, with a need for more transparent evidence of their efficacy, as well as self-doubt if the CR was ignored [59,63].
	Digital literacy and acceptability	Ability to understand, use, and incorporate digital technologies in health care.	Recent cohorts of GPs were more familiar and trusting of digitalization [59]. There were concerns that training, experience, and context would be subjugated to computer-generated recommendations [59,61].
	Patient-clinician relationship	The impact on patient interactions, communication, and consultation behaviors.	CRs could disrupt consultations and inhibit patient communication [61,63].
Organization
	Training in the purpose and use of CRs	Importance of providing adequate training and ensuring clinicians understand the purpose of CRs.	The importance of training was recognized when introducing CRs [58-61,63].
	Practice capacity and workflow	The capacity, workflow, and protocols of practice organizations.	Shortening consultation times precluded GPs from interacting with many of the CRs generated [58,60].
Wider system adapting and embedding over time
	Adaptation over time	The ability of CRs to incorporate technological advances and reflect the evolving needs and preferences of patients and clinicians.	The need for greater and earlier engagement of software developers with GPs was described [58,63,65]. A number of suggestions for future CR design were shared across studies including better integration with existing software systems, patient-specific content, clear graphical formats, and the ability to customize frequency and content [59-61,63,67].
	Systemwide collaboration	Unified approaches to the long-term introduction of technology including procurement, infrastructure, and shared learning.	There was a recognized need for improved interoperability and shared data between settings and organizations [63,66] and for continued learning from peer experience was voiced [59].

^aNASSS: Non-adoption, Abandonment, Scale-up, Spread, and Sustainability.

^bCR: clinical reminder.

^cGP: general practitioner.

Condition

The studies identified described how CRs were used to optimize care and manage patient safety in 3 key areas: medication [58,59,62-64,66]; diagnosis, both broadly [64,65] and for specific conditions, namely, atrial fibrillation and stroke [60], and cancer [61]. CRs were also used in the ongoing management of chronically ill patients, providing reminders of reviews or tests [64].

Technology Challenges in Technological Aspects of CRs: Alerts, Design, and Infrastructure

Seven studies described issues relating to various technological aspects of CRs, specifically the frequency and content of alerts, their design and integration with existing interfaces, and the broader digital infrastructure and capability including the interoperability of data.

Frequency and Content of CRs

Four studies described how the inability to process a large number of alerts was a concern among GPs [59,61,63,64]. As one GP described:

There are so many things (...) so many things popping up, so many things prompting you. You don’t probably respond to all the prompts, because there’s a box here, a box there, a box everywhere, and you don’t see everything (...) It’s such a busy screen you don’t respond to everything...

The value of presenting accompanying patient-centered information alongside the clinical data of the CR was described in one study. As a GP elaborated:

It [the CR] actually comes up with a nice diagram that it can use with your patients, that’s got smiley faces of different colours to illustrate risk (...) it’s just a different way to help augment that communication with them. For things like that it’s quite helpful because patients are wary of tablets, quite rightly.

Design and Integration of CRs

Three studies described how pop-up CRs that required the checking of drop-down lists or otherwise had to be actioned before any next step were overly disruptive and actually inhibited their engagement with the CRs content [59,62,63]:

[when they appear]...you can’t go looking in the notes, you can’t input anything else, this is now taking priority, so you either suspend it and then re-open it and suspend it and re-open it.

Digital Infrastructure

The capacity of practice information technology (IT) systems and hardware was also a consideration, for example, some CRs slowed the operating speed of practice computers [59,60,62,66].

Our problem here is our computers are rubbish. They work really, really slowly (...) it takes a long time to come through so we scroll down an option, a pick list - it can pick the wrong thing! Which is very frustrating...

Value Proposition GPs’ Perspectives on CR Quality, Safety, and Educational Impact

GPs described their perspectives on the quality and safety of CRs, their potential to support continuing education and training, and their impact on self-efficacy.

Overall Safety and Quality

In 5 studies GPs reported that CRs were capable of improving patient safety by promoting reflection on their decision [58-61,63]:

...even if you immediately dismiss it, at least that millisecond you’ve thought about it, and I think that is going to be useful at some point, but for how many people I don’t know

GPs also appreciated that CRs could improve the quality of care by providing useful reminders of tasks or actions [59,60]:

...in a patient with diabetes where there are 17 things to do and I’ve just missed out two or three of them (...) it’s really nice to have “somebody” say “Oh, don’t forget, there’s the urine test, and you haven’t yet sent the patient for the eye test that you thought somebody else was doing, and it looks like nobody’s done after all so maybe get that done?”

GPs in one study reported how CRs could improve how they practice in the long term by their ability to promote and reinforce appropriate prescribing behaviors.

...and it’s actually educational, because a lot of them you know the warning before it comes up because you’ve seen it so many times so you’re already aware of it. So I mean some of the things I know now are because of it, so there is an educational element to it [...]

Self-Efficacy

There were conflicting accounts regarding the extent to which CRs enhance GPs’ confidence in practicing safely. In one study, GPs described how the presence of CRs was a source of reassurance, supporting their ability to provide safe care in the increasingly complex environment of modern primary care [61,63].

...work in general practice is getting bigger and bigger, and more work’s going to come to general practice. So, although it [the CR] can cause irritation, the ﬂip side of that is you can relax a little bit more and that you don’t have to remember absolutely everything (...)

In another study, a GP reflected upon how having to choose a predetermined response outside of their original decision led them to question their ability [59]:

...in order to dismiss it you have to select what action you’ve taken such as, well have you called an ambulance, have you sent them into hospital...you’re then panicking thinking gosh I don’t want, I don’t want to lie, no I haven’t sent them into hospital but is that a judgement, is that them saying because I haven’t done that I’m a rubbish doctor?

Adopter Factors Influencing GPs’ Engagement With CRs: Digital Literacy and Consultation Impact

From the perspective of those adopting CRs, GPs described the factors affecting their engagement with CRs, the influence of digital literacy, and the impact of CRs on their consultation.

Engagement and Compliance in CRs

In one study, GPs expressed hesitation in relying on a computer-generated algorithm [59]. Concerns surfaced in 3 studies over the accuracy and relevance of the recommendations associated with CRs, with examples including unrecognized contraindications [58,63,65]:

[drug-drug interactions] were mentioned in printed textbooks but were not picked up by the interaction checking facility of their prescribing software

One study described a preference for greater transparency between the clinical evidence and the presentation of the CR. As one GP described:

[We’re] inherently distrustful of anything that hasn’t got somebody saying “Oh! I’ve done this trial, and this trial, and we’ve used it!” and I think that’s what we’ve been taught at medical school - to be sceptical, until we’ve got the evidence.

Digital Literacy and Acceptability

The belief emerged that the new generation of GPs more familiar with digital technology in everyday life would be more accepting of computer-generated reminders [59].

I think to a certain extent there is a bit of a cultural change in the sense that the newer generation of doctors coming through are much more used to computers telling them things in their own lives and so they’re much more accepting of the idea that the computer might give them helpful information.

GPs in 3 studies described how the reliance on CRs risked undermining the accrual of contextual knowledge of a particular patient [59,61,67].

A lot of the patients who I kind of see regularly with chronic problems, I kind of know if they’re well or not because I know the patient and I just can tell if they’re themselves or if they’re not because you have a constant, you know, you lose that with a computer.

Related to this was the feeling that CRs were overly prescriptive and one GP felt that the value of their input was diminished relative to the diktats of the CR:

I think it feels frustrating...when the system is so process-driven that there’s no sort of autonomy of the GP to, you know, change it or, kind of, put in what’s relevant.

Patient-Clinician Relationship

Two studies reported that GPs understood the importance of focusing on patients during consultations, acknowledging that referral to, and interaction with, CRs can be an impediment to patient-centered interaction [61,63]:

I don’t use any separate clinical decision support tool in the consultation because of time problems, losing eye contact with patients

Organization Organizational Influences on CR Adoption: Training and System Capacity

In considering organizational influences, evidence emerged of the importance of training and the impact of existing practice capacity and systems.

Training and Purpose

GPs in 5 studies reflected on the importance of training not only to understand the purpose of CRs but also their functionality [58-61,63]. One GP bemoaned the typical lack of training before the introduction of digital tools:

...so, yeah things just happen [. . .] They often will appear before you have any training or knowledge of it whatsoever. Then if you’re lucky maybe there’s a launch event six months after you’ve already started using it.

Practice Capacity and Workflow

GPs noted how the pressure on their time restricted their ability to interact with CRs that take time to resolve [58,60]. As one GP explained:

We might have six or seven of them [reminders], and once you’ve got (...) worked through about three or four, and then there’s this anticoagulation...you think, well that’s another 5 or 10 minutes of consultation (...) so you leave it.

In another example, one GP likened the appearance of a CR to a colleague bursting through the door during a consultation:

It really is like an interruption, you know, no GP wants somebody to just burst in with the door opening or the phone ringing. In the same vein no GP really wants a big thing to just pop up on the screen that they didn’t call up.

Wider System and Embedding Over Time Enhancing CR Implementation: Systemwide Collaboration and Adaptation Strategies

The potential impact of improved systemwide collaboration was described, and a number of suggestions were made as to how CRs may be adapted to ensure more sustained implementation.

Systemwide Collaboration

It was suggested in 2 studies that more closely integrated data sets across settings would support the quality of the CRs [63,66].

I want a dynamic electronic health communication with the possibility of a written dialog and forwarding missing information.

The importance of sharing best practices and recommendations for certain CR systems between peers was also described:

I think there is a degree of sharing these kind of protocols and algorithms amongst GP practises and we certainly, you know, GPs talk to each other and say “oh yeah we’ve got this thing that alerts for that particular problem”, “oh, have you? Right great can we, can we bring it in?”

Adaptation Over Time

Three studies described how GPs would have liked greater engagement with the software developers in the design of the CRs [58,63,65]. Suggestions for ways in which CRs might be improved included the acquisition of patient-specific alerts, better integration into practice workflows, and customizability informed by GP feedback [59-61,63]. In terms of their design and content, there were also suggestions for the inclusion of additional details on dosage or color-grading CRs according to severity [63].

Discussion General Findings

A total of 9 studies published since 2003 were identified, reporting research conducted in Europe, predominantly the United Kingdom, and Australia, 7 of which were judged to be of high quality. The findings from these studies were effectively categorized within the domains of the NASSS framework. Regarding Condition and Value Proposition, the use of CRs in diagnosis, prescribing, and the ongoing management of chronic conditions was seen as an effective means of maintaining or improving safe and high-quality care, albeit with contrasting effects on clinician self-efficacy. In terms of Technology attributes, the efficacy of CRs was found to be directly linked to their frequency, the presentation and accuracy of their content, and the speed and reliability of IT infrastructure and hardware. With respect to Adopter attitudes, GPs raised concerns about the accuracy of CRs and the potential loss of benefits derived from their clinical experience and consideration of individual patient contexts, in favor of algorithm-based decisions. From an Organization perspective, the importance of training and the ability of CRs to integrate seamlessly with existing workflows were emphasized. Finally, in the context of the Wider System, several suggestions were made to improve the usability of CRs, including the sharing of best practices and involving GPs earlier in the design process to ensure successful embedding over time.

Specific Findings and Comparison With Existing Literature Condition/Value Proposition

Managing the medications of a growing number of patients with multiple long-term conditions is a major contributor to GP workload in the United Kingdom and other countries [68,69]. The studies we identified suggest that CRs can play a positive role in supporting this management [58,59,62-64,66], and it is widely recognized that CRs present a valuable opportunity to reduce adverse drug reactions [70].

The potential of CRs to support diagnosis was highlighted in 4 of the identified studies [60,61,64,65]. This ability has been previously acknowledged as a significant advantage in primary care, where diagnostic errors are a critical concern, particularly for diseases with low incidence but high health risk [71-73].

Primary care is expected to take on an increasing role in managing chronic conditions across various countries and health systems, including the United Kingdom [74,75]. However, this capacity is often hindered by limited consultation times, high patient volumes, and inadequate communication with specialists [75,76]. In such contexts, CRs that draw on patient-specific data to remind GPs to test, review, or conduct periodic interventions appear highly valuable, though only 1 study explicitly examined this functionality [64].

The ability of CRs to prompt reflection on clinical decision-making was viewed positively in 5 of the studies [58-61,63]. The benefits of reflective practice on clinical decisions are well established, particularly when the feedback on decisions, as with CRs, is immediate [77].

Technology

The studies we identified confirmed that GPs felt overwhelmed by the number, complexity, and often poor functionality of CRs [59,61,63,64]. This cognitive overload caused by multiple appearances of CRs reflects previous research suggesting it can hinder clinician interaction with clinical decision support systems [18-20,78]. The cognitive burden associated with CRs also stems from the complexity of their content [59,62,63]. Therefore, CRs should be designed to accurately convey risk and relevant information in a clear, easily interpretable format that adheres to best practices in digital interface design [79-82]. Additionally, they should be better integrated into existing software and workflows [83,84].

Artificial intelligence has begun to be utilized in clinical decision support systems for individual conditions, and it is reasonable to assume it will play a role in refining the specificity of CRs in primary care [85]. However, efforts to minimize the negative impact of CRs, such as improving specificity [18] or allowing clinicians the autonomy to disable certain CRs [83], must remain mindful of patient safety implications [86].

The ability of GPs to respond promptly to CRs was also influenced by the speed and reliability of practice IT infrastructure and hardware [59,60,62,66]. Ensuring that existing IT systems can handle the growing prevalence and complexity of digital health care is widely acknowledged as a critical need in both high- and low- and middle-income countries. However, this challenge has yet to be universally addressed [87,88].

Adopter

GP concerns about CRs that enforce process-driven decisions at the expense of context and experience were described in 4 of the studies we identified [58,59,63,65]. Notably, lasting cynicism expressed by GPs who had experienced prior failures with CRs (described in 2 of the identified studies [59,63]) has been previously recognized in primary care, where digital interventions have failed to meet expectations [89].

A framework for technology developers has emerged that aims to enhance clinician confidence in clinical decision support systems by adhering to a predetermined set of criteria, focusing on understanding and sustaining trust among end users [90,91]. Evidence from one study we identified suggested that resistance to digital interventions may be lower among the generation of clinicians who grew up alongside the proliferation of digital technologies [59]. It is hoped that trust in CRs can be fostered through greater and earlier engagement with clinician end users by software developers and senior decision makers [92-94]. This aligns with long-standing recommendations for increased user involvement in the development of digital health tools [67,89].

The negative disruption of patient consultations by CRs was also highlighted [61,63]. It is already known that technology can interrupt the flow of conversation, reduce nonverbal communication cues, and compromise the overall quality of patient interactions [95,96]. Concerns expressed by GP participants in the identified studies reflect existing evidence that clinicians highly value uninterrupted in-person consultations, as they build patient trust and foster a constructive provider-patient relationship [97,98].

Organization

The lack of training emerged as a barrier to engagement and compliance with CRs in the studies we identified [58-61,63] and has previously been highlighted as central to their effective use, including sharing with GPs the processes behind their generation [99,100]. However, targeted training to enhance engagement with CRs has yet to be systematically implemented in the high-income countries where they are most prevalent [101].

The pressure on GPs to address increasingly complex care needs and rising patient expectations within shortening consultation times is widely acknowledged [102]. GPs in the identified studies reported that managing multiple CRs was unfeasible within the constraints of limited consultation times [58,60]. Some studies suggest that specific CRs, such as those supporting preventive care, could be delivered to patients before the consultation to alleviate the time pressure on GPs [103].

Wider System and Embedding Over Time

The need for systemwide coordination and collaboration in introducing CRs into primary care, as described in the review’s findings [59,63,66], has also been recognized in other health care settings [104-106]. Similarly, the calls for earlier clinician engagement in the design and development of CRs identified in this review [58,63,65] have been previously acknowledged [107-109], alongside the input of clinical informaticists [110]. While there are recognized challenges in engaging time-pressured GPs during the early stages of software development, sustainable adoption of health technology requires understanding its social and organizational contexts. This can only be achieved by involving end users throughout its design and development [111,112].

Strengths and Limitations

The recognized systematic review methodology was applied to identify relevant papers [27,113], including the dual-review process [114]. The comprehensive NASSS framework successfully accommodated all of the data identified [26] and its use demonstrated both the transparency and rigor of our approach but also allowed us to structure and contextualize our data with existing research to avoid distorting our conclusions [28]. We acknowledge that there were a number of potential limitations. First, all the studies identified originated in high-income countries and were presented in English; however, we would suggest that the lessons learned have the potential for broader applicability for digital interventions in health systems of low- and middle-income countries that in reality share many of the same, if accentuated, barriers to technology-enabled care [111,112]. Second, it is possible that the use of an a priori framework might limit the analysis, with some data unable to be allocated within its confines; however, taking the unconstrained matrix approach suggested by Elo and Kyngäs [56] allowed the development and inclusion of emergent themes within the established framework. This meant we could maintain alignment with our established objectives and provide a systematic description of novel findings.

Conclusions

If utilized correctly CRs can enhance patient safety, workflow management, preventative care, and reduce cognitive overload. Based on the findings of the review several recommendations can be made for the future design and implementation of CRs (see Table 4). These include improved graphical designs better suited to the purpose of the CR, more nuanced and patient-specific content, greater clinician autonomy regards if and when CRs are seen, and earlier and more consistent engagement with the workforce in their design and development.

Table 4

Recommendations for future CR^a design and implementation.

Domain	Recommendation
Technology	Visual-driven user interface to present information more effectively and reduce information overload. More directly applicable data (eg, dosage or risk). The capacity of the infrastructure.
Adopter	GP^b-endorsed or GP-recommended CRs would promote and increase their uptake. Ensuring that CRs transparently reflect the latest clinical evidence.
Organization	Greater engagement with the workforce in the introduction of CRs. Additional training would help the uptake of CRs and using them effectively, accommodating the age and experience of the clinician. Enable organizations to better incorporate CRs in existing/novel workflows.
Wider systems	GP-stakeholders collaboration would enable feedback and co-design of alert utilization.
Adaptation	Alert customization would allow GPs to deactivate unnecessary options to change output and will provide flexibility.

^aCR: clinical reminder.

^bGP: general practitioner.

Multimedia Appendix 1

Search strategy.

Multimedia Appendix 2

Eligibility criteria.

Multimedia Appendix 3

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 checklist (v17.12.24).

Multimedia Appendix 4

Study selection and data extraction templates.

Multimedia Appendix 5

Excluded studies.

Multimedia Appendix 6

Quality appraisal.

Abbreviations

clinical reminder

general practitioner

information technology

NASSS

Non-adoption, Abandonment, Scale-up, Spread, and Sustainability

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

The main funding for this project was provided by the Engineering and Physical Sciences Research Council (grant EP/M014401).

Data Availability

This is a systematic review and the data it uses are in the public domain.

IL was responsible for conceptualization, funding acquisition, methodology, project administration, supervision, formal analysis, writing—original draft, writing/review, and editing. IG was responsible for the methodology, investigation, data curation, formal analysis, writing—original draft, and writing—review & editing. DS was responsible for validation and writing—review and editing. GD was responsible for writing—review and editing. NC was responsible for writing—review and editing. AP was responsible for data curation, formal analysis, validation, and writing—review and editing.

None declared.

Macdonald

Clements

Sobti

Rossiter

Unnithan

Bosanquet

Tackling the elective case backlog generated by Covid-19: the scale of the problem and solutions

J Public Health (Oxf) 2020 11 23 42 4 712 716

10.1093/pubmed/fdaa155

32930797

5905817

PMC7543523

Henley

Connolly

Jones

Giuffrida

A ticking time bomb?: Health care under threat across western Europe

The Guardian 2022

2024-07-01

https://www.theguardian.com/society/2022/dec/14/a-ticking-time-bomb-healthcare-under-threat-across-western-europe

Grimshaw

Thomas

MacLennan

Fraser

Ramsay

Vale

Whitty

Eccles

Matowe

Shirran

Wensing

Dijkstra

Donaldson

Effectiveness and efficiency of guideline dissemination and implementation strategies

Int J Technol Assess Health Care 2005 03 02 21 1 149 149

10.1017/s0266462305290190

Grimshaw

Eccles

Tetroe

Implementing clinical guidelines: current evidence and future implications

J Contin Educ Health Prof 2004 24 Suppl 1 S1 S31 7

10.1002/chp.1340240506

15712775

Brokel

Schwichtenberg

Wakefield

Ward

Shaw

Kramer

Evaluating clinical decision support rules as an intervention in clinician workflows with technology

Computers, Informatics, Nursing 2011 29 1 36 42

10.1097/ncn.0b013e3181f9dbb1

Ancker

Edwards

Nosal

Hauser

Mauer

Kaushal

with the HITEC Investigators

Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system

BMC Med Inform Decis Mak 2017 04 10 17 1 36 2

10.1186/s12911-017-0430-8

28395667

10.1186/s12911-017-0430-8

PMC5387195

Saleem

Patterson

Militello

Anders

Falciglia

Wissman

Roth

Asch

Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses

Journal of the American Medical Informatics Association 2007 09 01 14 5 632 640

10.1197/jamia.m2163

Ivers

Jamtvedt

Flottorp

Young

Odgaard-Jensen

Jan

French

O'Brien

Mary Ann

Johansen

Marit

Grimshaw

Jeremy

Oxman

Andrew D

Audit and feedback: effects on professional practice and healthcare outcomes

Cochrane Database Syst Rev 2012 06 13 2012 6 CD000259

10.1002/14651858.CD000259.pub3

22696318

PMC11338587

Baker

Camosso-Stefinovic

Janette

Gillies

Shaw

Cheater

Flottorp

Robertson

Noelle

Wensing

Michel

Fiander

Michelle

Eccles

Martin P

Godycki-Cwirko

Maciek

van Lieshout

Jan

Jäger

Cornelia

Tailored interventions to address determinants of practice

Cochrane Database Syst Rev 2015 04 29 2015 4 CD005470

10.1002/14651858.CD005470.pub3

25923419

PMC7271646

Dexheimer

Talbot

Sanders

Rosenbloom

Aronsky

Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials

Journal of the American Medical Informatics Association 2008 05 01 15 3 311 320

10.1197/jamia.m2555

Cecil

Dewa

Majeed

Aylin

General practitioner and nurse practitioner attitudes towards electronic reminders in primary care: a qualitative analysis

BMJ Open 2021 07 12 11 7 e045050

10.1136/bmjopen-2020-045050

34253661

bmjopen-2020-045050

PMC8276294

Moxey

Robertson

Newby

Hains

Williamson

Pearson

Sallie-Anne

Computerized clinical decision support for prescribing: provision does not guarantee uptake

J Am Med Inform Assoc 2010 17 1 25 33

10.1197/jamia.M3170

20064798

17/1/25

PMC2995634

Litvin

Davis

Moran

Iverson

Zhao

Zapka

The use of clinical decision‐support tools to facilitate geriatric education

J American Geriatrics Society 2012 05 29 60 6 1145 1149

10.1111/j.1532-5415.2012.03960.x

Trevena

McCaffery

Salkeld

Glasziou

Del

Doust

Clinical decision-making tools: how effective are they in improving the quality of health care?

Analysis & Policy Observatory 2014

2025-01-30

https://apo.org.au/node/53712

Elwyn

Scholl

Tietbohl

Mann

Edwards

Clay

Légaré

Weijden

TVD

Lewis

Wexler

Frosch

“Many miles to go …”: a systematic review of the implementation of patient decision support interventions into routine clinical practice

BMC Med Inform Decis Mak 2013 11 29 13 S2 e

10.1186/1472-6947-13-s2-s14

Magnus

Rodgers

Avery

GPs' views on computerized drug interaction alerts: questionnaire survey

J Clin Pharm Ther 2002 10 27 5 377 82

10.1046/j.1365-2710.2002.00434.x

12383140

434

Militello

Patterson

Tripp-Reimer

Asch

Fung

Glassman

Anders

Doebbeling

Clinical reminders: why don't they use them?

Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2004 09 01

The Human Factors and Ergonomics Society Annual Meeting

September 1, 2004

New Orleans, LO

1651 1655

10.1177/154193120404801505

van der Sijs

Aarts

Vulto

Berg

Overriding of drug safety alerts in computerized physician order entry

Journal of the American Medical Informatics Association 2006 03 01 13 2 138 147

10.1197/jamia.m1809

Weingart

Toth

Sands

Aronson

Davis

Phillips

Physicians' decisions to override computerized drug alerts in primary care

Arch Intern Med 2003 11 24 163 21 2625 31

10.1001/archinte.163.21.2625

14638563

163/21/2625

Hsieh

Kuperman

Jaggi

Hojnowski-Diaz

Fiskio

Williams

Bates

Gandhi

Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system

J Am Med Inform Assoc 2004 11 01 11 6 482 491

10.1197/jamia.m1556

Feldstein

Simon

Schneider

Krall

Laferriere

Smith

Sittig

Soumerai

How to Design Computerized Alerts to Ensure Safe Prescribing Practices

The Joint Commission Journal on Quality and Safety 2004 11 30 11 602 613

10.1016/s1549-3741(04)30071-7

15565759

S1549-3741(04)30071-7

Bell

Garfield

Khosla

Patel

Franklin

Mixed methods study of medication-related decision support alerts experienced during electronic prescribing for inpatients at an English hospital

Eur J Hosp Pharm 2019 11 19 26 6 318 322

10.1136/ejhpharm-2017-001483

31798854

ejhpharm-2017-001483

PMC6855857

Thomas Craig

Kelly J

Willis

Gruen

Rhee

Jackson

The burden of the digital environment: a systematic review on organization-directed workplace interventions to mitigate physician burnout

J Am Med Inform Assoc 2021 04 23 28 5 985 997

10.1093/jamia/ocaa301

33463680

6104341

PMC8068437

Teich

Osheroff

Pifer

Sittig

Clinical Decision Support in Electronic Prescribing: Recommendations and an Action Plan

J Am Med Inform Assoc 2005 07 01 12 4 365 376

10.1197/jamia.m1822

Saleem

Patterson

Militello

Render

Orshansky

Asch

Exploring barriers and facilitators to the use of computerized clinical reminders

J Am Med Inform Assoc 2005 07 01 12 4 438 447

10.1197/jamia.m1777

Greenhalgh

Wherton

Papoutsi

Lynch

Hughes

A'Court

Hinder

Fahy

Procter

Shaw

Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies

J Med Internet Res 2017 11 01 19 11 e367

10.2196/jmir.8775

29092808

v19i11e367

PMC5688245

Assarroudi

Heshmati Nabavi

Armat

Ebadi

Vaismoradi

Directed qualitative content analysis: the description and elaboration of its underpinning methods and data analysis process

J Res Nurs 2018 03 10 23 1 42 55

10.1177/1744987117741667

34394406

10.1177_1744987117741667

PMC7932246

McKenzie

Brennan

Higgins

Thomas

Chandler

Cumpston

Higgins

JPT

Thomas

Chandler

Cumpston

Page

Welch

Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis

Cochrane Handbook for Systematic Reviews of Interventions version 6.5 2022

London, UK

Cochrane

Abimbola

Patel

Peiris

Patel

Harris

Usherwood

Greenhalgh

The NASSS framework for ex post theorisation of technology-supported change in healthcare: worked example of the TORPEDO programme

BMC Med 2019 12 30 17 1 233

10.1186/s12916-019-1463-x

31888718

10.1186/s12916-019-1463-x

PMC6937726

Litchfield

Glasby

Parkinson

Hocking

Tanner

Roe

Bousfield

“Trying to find people to fit the tech…”: a qualitative exploration of the lessons learnt introducing artificial intelligence-based technology into English social care

Health & Social Care in the Community 2023 5 16 2023 1 11

10.1155/2023/9174873

Greenhalgh

Maylor

Shaw

Wherton

Papoutsi

Betton

Nelissen

Gremyr

Rushforth

Koshkouei

Taylor

The NASSS-CAT tools for understanding, guiding, monitoring, and researching technology implementation projects in health and social care: protocol for an evaluation study in real-world settings

JMIR Res Protoc 2020 05 13 9 5 e16861

10.2196/16861

32401224

v9i5e16861

PMC7254278

Schaink

Kuluski

Lyons

Fortin

Jadad

Upshur

Wodchis

A scoping review and thematic classification of patient complexity: offering a unifying framework

J Comorb 2012 10 10 2 1 1 9

10.15256/joc.2012.2.15

29090137

PMC5556402

Proctor

Ramsey

Saldana

Maddox

Chambers

Brownson

FAST: a framework to assess speed of translation of health innovations to practice and policy

Glob Implement Res Appl 2022 06 02 2 2 107 119

10.1007/s43477-022-00045-4

35669171

PMC9161655

Schoville

Rhonda R

Titler

Marita G

Guiding healthcare technology implementation: a new integrated technology implementation model

Comput Inform Nurs 2015 03 33 3 99 107; quiz E1

10.1097/CIN.0000000000000130

25799235

00024665-201503000-00004

Belay

Desta

Smithson

Meshesha

Investigate knowledge management technology implementation for supporting decision making in Ethiopian health sectors

BMC Med Inform Decis Mak 2021 05 05 21 1 146

10.1186/s12911-021-01507-9

33952240

10.1186/s12911-021-01507-9

PMC8097940

Priestman

Collins

Vigne

Sridharan

Seamer

Bowen

Sebire

Lessons learned from a comprehensive electronic patient record procurement process-implications for healthcare organisations

BMJ Health Care Inform 2019 05 09 26 1 e000020

10.1136/bmjhci-2019-000020

31072821

bmjhci-2019-000020

PMC7062322

Terry

Appification, AI, and healthcare's new iron triangle

J Health Care L & Pol'y 2017 20 117

Lehoux

Miller

Daudelin

Denis

Providing value to new health technology: the early contribution of entrepreneurs, investors, and regulatory agencies

Int J Health Policy Manag 2017 09 01 6 9 509 518

10.15171/ijhpm.2017.11

28949463

PMC5582437

van Limburg

van Gemert-Pijnen

Nijland

Ossebaard

Hendrix

Seydel

Why business modeling is crucial in the development of eHealth technologies

J Med Internet Res 2011 12 28 13 4 e124

10.2196/jmir.1674

22204896

v13i4e124

PMC3278110

Perski

Short

Acceptability of digital health interventions: embracing the complexity

Transl Behav Med 2021 07 29 11 7 1473 1480

10.1093/tbm/ibab048

33963864

6272497

PMC8320880

Rahman

Warren

Carpenter

Healthcare Technology Self-Efficacy (HTSE) and its influence on individual attitude: an empirical study

Computers in Human Behavior 2016 05 58 12 24

10.1016/j.chb.2015.12.016

Lythreatis

Singh

El-Kassar

The digital divide: a review and future research agenda

Technological Forecasting and Social Change 2022 02 175 121359

10.1016/j.techfore.2021.121359

Caccia-Bava

Maria do Carmo

Guimaraes

Harrington

Hospital organization culture, capacity to innovate and success in technology adoption

J Health Organ Manag 2006 20 2-3 194 217

10.1108/14777260610662735

16869354

Cresswell

Sheikh

Franklin

Krasuska

Nguyen

Hinder

Lane

Mozaffar

Mason

Eason

Potts

HWW

Williams

Theoretical and methodological considerations in evaluating large-scale health information technology change programmes

BMC Health Serv Res 2020 05 27 20 1 477

10.1186/s12913-020-05355-7

32460830

10.1186/s12913-020-05355-7

PMC7254705

Fitzgerald

McDermott

Challenging Perspectives on Organizational Change in Health Care 2017

Abingdon, Oxon

Routledge

May

Finch

Implementing, embedding, and integrating practices: an outline of normalization process theory

Sociology 2009 06 15 43 3 535 554

10.1177/0038038509103208

Frønsdal

Facey

Klemp

Norderhaug

Mørland

Røttingen

Health technology assessment to optimize health technology utilization: using implementation initiatives and monitoring processes

Int J Technol Assess Health Care 2010 06 29 26 3 309 316

10.1017/s0266462310000309

Ofori-Asenso

Hallgreen

De Bruin

Improving interactions between health technology assessment bodies and regulatory agencies: a systematic review and cross-sectional survey on processes, progress, outcomes, and challenges

Front Med (Lausanne) 2020 10 16 7 582634

10.3389/fmed.2020.582634

33178721

PMC7596325

Greenhalgh

Russell

Why do evaluations of eHealth programs fail? An alternative set of guiding principles

PLoS Med 2010 11 02 7 11 e1000360

10.1371/journal.pmed.1000360

21072245

PMC2970573

Greenhalgh

Robert

Macfarlane

Bate

Kyriakidou

Diffusion of innovations in service organizations: systematic review and recommendations

Milbank Quarterly 2004 12 09 82 4 581 629

10.1111/j.0887-378x.2004.00325.x

Holden

Karsh

The technology acceptance model: its past and its future in health care

J Biomed Inform 2010 02 43 1 159 72

10.1016/j.jbi.2009.07.002

19615467

S1532-0464(09)00096-3

PMC2814963

Hollnagel

Braithwaite

Wears

Delivering Resilient Health Care 2018

Abingdon, Oxon

Routledge

Cooke

Smith

Booth

Beyond PICO: the SPIDER tool for qualitative evidence synthesis

Qual Health Res 2012 10 24 22 10 1435 43

10.1177/1049732312452938

22829486

1049732312452938

Backman

Ruth

Bayliss

Susan

Moore

David

Litchfield

Ian

Clinical reminder alert fatigue in healthcare: a systematic literature review protocol using qualitative evidence

Syst Rev 2017 12 13 6 1 255

10.1186/s13643-017-0627-z

29237488

10.1186/s13643-017-0627-z

PMC5729261

Hong

Fàbregues

Bartlett

Boardman

Cargo

Dagenais

Gagnon

Griffiths

Nicolau

O’Cathain

Rousseau

Vedel

Pluye

The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers

EFI 2018 12 18 34 4 285 291

10.3233/efi-180221

Elo

Kyngäs

Helvi

The qualitative content analysis process

J Adv Nurs 2008 04 18 62 1 107 15

10.1111/j.1365-2648.2007.04569.x

18352969

JAN4569

Bingham

Witkowsky

Deductive and inductive approaches

Analyzing and Interpreting Qualitative Data: After the Interview 2021

Thousand Oaks, CA

Sage Publications

Jeffries

Salema

Laing

Shamsuddin

Sheikh

Avery

Chuter

Waring

Keers

The implementation, use and sustainability of a clinical decision support system for medication optimisation in primary care: a qualitative evaluation

PLoS One 2021 5 3 16 5 e0250946

10.1371/journal.pone.0250946

33939750

PONE-D-20-35487

PMC8092789

Ford

Edelman

Somers

Shrewsbury

Lopez Levy

van Marwijk

Curcin

Porat

Barriers and facilitators to the adoption of electronic clinical decision support systems: a qualitative interview study with UK general practitioners

BMC Med Inform Decis Mak 2021 06 21 21 1 193

10.1186/s12911-021-01557-z

34154580

10.1186/s12911-021-01557-z

PMC8215812

Holt

Dalton

Kirkpatrick

Hislop

Marshall

Fay

Qureshi

Lasserson

Kearley

Mollison

Fitzmaurice

Hobbs

Barriers to a software reminder system for risk assessment of stroke in atrial fibrillation: a process evaluation of a cluster randomised trial in general practice

Br J Gen Pract 2018 11 05 68 677 e844 e851

10.3399/bjgp18x699809

Dikomitis

Green

Macleod

Embedding electronic decision-support tools for suspected cancer in primary care: a qualitative study of GPs’ experiences

Prim Health Care Res Dev 2015 3 3 16 06 548 555

10.1017/s1463423615000109

Slight

Howard

Ghaleb

Barber

Franklin

Avery

The causes of prescribing errors in English general practices: a qualitative study

Br J Gen Pract 2013 10 01 63 615 e713 e720

10.3399/bjgp13x673739

Ahearn

Kerr

General practitioners' perceptions of the pharmaceutical decision-support tools in their prescribing software

Med J Aust 2003 07 07 179 1 34 7

10.5694/j.1326-5377.2003.tb05415.x

12831382

ahe10824_fm

Heselmans

Aertgeerts

Donceel

Geens

Van de Velde

Ramaekers

Family physicians' perceptions and use of electronic clinical decision support during the first year of implementation

J Med Syst 2012 12 9 36 6 3677 84

10.1007/s10916-012-9841-3

22402980

Bindels

Hasman

Arie

Derickx

Mieke

Van Wersch

Jan W J

Winkens

Ron A G

User satisfaction with a real-time automated feedback system for general practitioners: a quantitative and qualitative study

Int J Qual Health Care 2003 12 01 15 6 501 8

10.1093/intqhc/mzg076

14660533

Christensen

Grimsmo

Expectations for the next generation of electronic patient records in primary care: a triangulated study

Inform Prim Care 2008 16 1 21 8

10.14236/jhi.v16i1.671

18534074

671

Mahmud

Islam

Ahmed

Smolander

What influences algorithmic decision-making? A systematic literature review on algorithm aversion

Technological Forecasting and Social Change 2022 02 175 121390

10.1016/j.techfore.2021.121390

Carrier

Zaytseva

Bocquier

Villani

Verdoux

Fortin

Verger

GPs’ management of polypharmacy and therapeutic dilemma in patients with multimorbidity: a cross-sectional survey of GPs in France

Br J Gen Pract 2019 02 25 69 681 e270 e278

10.3399/bjgp19x701801

Reeve

Avoiding harm: Tackling problematic polypharmacy through strengthening expert generalist practice

Brit J Clinical Pharma 2020 09 09 87 1 76 83

10.1111/bcp.14531

Armando

Miglio

de Cosmo

Cena

Clinical decision support systems to improve drug prescription and therapy optimisation in clinical practice: a scoping review

BMJ Health Care Inform 2023 05 02 30 1 e100683

10.1136/bmjhci-2022-100683

37130626

bmjhci-2022-100683

PMC10163516

Lehtovuori

Heikkinen

Raina

Kauppila

The effect of electronic reminders on the recording of diagnoses in primary care: a quasi-experimental before and after study

SAGE Open Med 2020 04 28 8 2050312120918267

10.1177/2050312120918267

32435481

10.1177_2050312120918267

PMC7222644

Singh

Schiff

Graber

Onakpoya

Thompson

The global burden of diagnostic errors in primary care

BMJ Qual Saf 2017 06 16 26 6 484 494

10.1136/bmjqs-2016-005401

27530239

bmjqs-2016-005401

PMC5502242

Round

Steed

Shankleman

Bourke

Risi

Primary care delays in diagnosing cancer: what is causing them and what can we do about them?

J R Soc Med 2013 11 09 106 11 437 40

10.1177/0141076813504744

24108536

0141076813504744

PMC3807772

Nolte

Knai

Saltman

Assessing chronic disease management in European health systems

European Observatory on Health Systems and Policies 2014

Copenhagen, Denmark

European Observatory on Health Systems and Policies

Reynolds

Dennis

Hasan

Slewa

Chen

Tian

Bobba

Zwar

A systematic review of chronic disease management interventions in primary care

BMC Fam Pract 2018 01 09 19 1 11

10.1186/s12875-017-0692-3

29316889

10.1186/s12875-017-0692-3

PMC5759778

Foo

Sundram

Legido-Quigley

Facilitators and barriers of managing patients with multiple chronic conditions in the community: a qualitative study

BMC Public Health 2020 02 27 20 1 273

10.1186/s12889-020-8375-8

32106838

10.1186/s12889-020-8375-8

PMC7045577

Prakash

Sladek

Schuwirth

Interventions to improve diagnostic decision making: a systematic review and meta-analysis on reflective strategies

Medical Teacher 2018 09 23 41 5 517 524

10.1080/0142159x.2018.1497786

Westerbeek

Ploegmakers

de Bruijn

Linn

van Weert

Daams

van der Velde

van Weert

Abu-Hanna

Medlock

Barriers and facilitators influencing medication-related CDSS acceptance according to clinicians: a systematic review

Int J Med Inform 2021 08 152 104506

10.1016/j.ijmedinf.2021.104506

34091146

S1386-5056(21)00132-5

Horsky

Schiff

Johnston

Mercincavage

Bell

Middleton

Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions

J Biomed Inform 2012 12 45 6 1202 16

10.1016/j.jbi.2012.09.002

22995208

S1532-0464(12)00149-9

Wiklund

Kendler

Hochberg

Weinger

Technical basis for user interface design of health IT

National Institute of Standards and Technology (NIST) 2015

2025-01-30

https://nvlpubs.nist.gov/nistpubs/gcr/2015/NIST.GCR.15-996.pdf

Silva

Sousa Santos

Madeira

Using color in visualization: a survey

Computers & Graphics 2011 04 35 2 320 333

10.1016/j.cag.2010.11.015

Franconeri

Padilla

Shah

Zacks

Hullman

The science of visual data communication: what works

Psychol Sci Public Interest 2021 12 15 22 3 110 161

10.1177/15291006211051956

34907835

Olakotan

Mohd Yusof

The appropriateness of clinical decision support systems alerts in supporting clinical workflows: a systematic review

Health Informatics J 2021 04 15 27 2 14604582211007536

10.1177/14604582211007536

33853395

Sharma

Bhaskar

Addressing the Covid-19 burden on medical education and training: the role of telemedicine and tele-education during and beyond the pandemic

Front Public Health 2020 11 27 8 589669

10.3389/fpubh.2020.589669

33330333

PMC7728659

Wang

Chen

Zhang

Huang

Sun

Yuan

Artificial intelligence in clinical decision support systems for oncology

Int J Med Sci 2023 20 1 79 86

10.7150/ijms.77205

36619220

ijmsv20p0079

PMC9812798

Kesselheim

Cresswell

Phansalkar

Bates

Sheikh

Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation

Health Aff (Millwood) 2011 12 30 12 2310 7

10.1377/hlthaff.2010.1111

22147858

30/12/2310

Abernethy

Adams

Barrett

Bechtel

Brennan

Butte

Faulkner

Judith

Fontaine

Elaine

Friedhoff

Stephen

Halamka

John

Howell

Michael

Johnson

Kevin

Long

Peter

McGraw

Deven

Miller

Redonda

Lee

Peter

Perlin

Jonathan

Rucker

Donald

Sandy

Lew

Savage

Lucia

Stump

Lisa

Tang

Paul

Topol

Eric

Tuckson

Reed

Valdes

Kristen

The promise of digital health: then, now, and the future

NAM Perspect 2022 2022 10.31478/202206e

10.31478/202206e

36177208

10.31478/202206e

PMC9499383

Labrique

Wadhwani

Williams

Lamptey

Hesp

Luk

Aerts

Best practices in scaling digital health in low and middle income countries

Global Health 2018 11 03 14 1 103

10.1186/s12992-018-0424-z

30390686

10.1186/s12992-018-0424-z

PMC6215624

Ziebland

Hyde

Powell

Power, paradox and pessimism: On the unintended consequences of digital health technologies in primary care

Soc Sci Med 2021 11 289 114419

10.1016/j.socscimed.2021.114419

34619631

S0277-9536(21)00751-6

Jones

Thornton

Wyatt

Enhancing trust in clinical decision support systems: a framework for developers

BMJ Health Care Inform 2021 06 04 28 1 e100247

10.1136/bmjhci-2020-100247

34088721

bmjhci-2020-100247

PMC8183267

O’Neill

Linking Trust to Trustworthiness

International Journal of Philosophical Studies 2018 04 25 26 2 293 300

10.1080/09672559.2018.1454637

McDermott

Checkland

Coleman

Osipovič

Petsoulas

Perkins

Engaging GPs in commissioning: realist evaluation of the early experiences of Clinical Commissioning Groups in the English NHS

J Health Serv Res Policy 2017 01 08 22 1 4 11

10.1177/1355819616648352

27151153

1355819616648352

PMC5207294

Duncan

Ridd

McCahon

Guthrie

Cabral

Barriers and enablers to collaborative working between GPs and pharmacists: a qualitative interview study

Br J Gen Pract 2020 02 10 70 692 e155 e163

10.3399/bjgp20x708197

Sherrill

Wiese

Abdullah

Arriaga

Overcoming clinician technophobia: what we learned from our mass exposure to telehealth during the COVID-19 pandemic

J Technol Behav Sci 2022 08 19 7 4 547 553

10.1007/s41347-022-00273-3

36034538

273

PMC9391067

Kumarapeli

de Lusignan

Using the computer in the clinical consultation; setting the stage, reviewing, recording, and taking actions: multi-channel video study

J Am Med Inform Assoc 2013 06 01 20 e1 e67 75

10.1136/amiajnl-2012-001081

23242763

amiajnl-2012-001081

PMC3715353

Nouri

Khoong

Lyles

Karliner

Addressing equity in telemedicine for chronic disease management during the Covid-19 pandemic

NEJM Catalyst 2020 1 3 e1

Caldwell

The process of clinical consultation is crucial to patient outcomes and safety: 10 quality indicators

Clin Med (Lond) 2019 11 19 6 503 506

10.7861/clinmed.2019-0263

31732592

S1470-2118(24)00106-4

PMC6899233

Crook

Iftikhar

Moore

Lowdon

Modarres

Message

A comparison of in-person versus telephone consultations for outpatient hospital care

Future Healthc J 2022 07 9 2 154 160

10.7861/fhj.2022-0006

35928204

S2514-6645(24)00492-2

PMC9345241

Kersting

Weltermann

Electronic reminders to facilitate longitudinal care: a mixed-methods study in general practices

BMC Med Inform Decis Mak 2016 11 24 16 1 148

10.1186/s12911-016-0387-z

27881130

10.1186/s12911-016-0387-z

PMC5122020

100

Khairat

Marc

Crosby

Al Sanousi

Reasons for physicians not adopting clinical decision support systems: critical analysis

JMIR Med Inform 2018 04 18 6 2 e24

10.2196/medinform.8912

29669706

v6i2e24

PMC5932331

101

Johnson

Zheng

Padman

Modeling the longitudinality of user acceptance of technology with an evidence-adaptive clinical decision support system

Decision Support Systems 2014 01 57 444 453

10.1016/j.dss.2012.10.049

102

Salisbury

Helen Salisbury: the 10 minute appointment

BMJ 2019 06 04 365 l2389

10.1136/bmj.l2389

31164335

103

Frank

Clinical software packages can be improved to help general practitioners increase the provision of preventive care

Aust. J. Prim. Health 2022 4 28 28 3 204 206

10.1071/py21204

104

Sutton

Pincock

Baumgart

Sadowski

Fedorak

Kroeker

An overview of clinical decision support systems: benefits, risks, and strategies for success

NPJ Digit Med 2020 02 06 3 1 17

10.1038/s41746-020-0221-y

32047862

221

PMC7005290

105

Chen

O'Bryan

Claire Maree

Gorham

Howard

Balasubramanya

Coffey

Abeyaratne

Cass

Barriers and enablers to implementing and using clinical decision support systems for chronic diseases: a qualitative systematic review and meta-aggregation

Implement Sci Commun 2022 07 28 3 1 81

10.1186/s43058-022-00326-x

35902894

10.1186/s43058-022-00326-x

PMC9330991

106

Meunier

Raynaud

Guimaraes

Gueyffier

Letrilliart

Barriers and facilitators to the use of clinical decision support systems in primary care: a mixed-methods systematic review

Ann Fam Med 2023 01 23 21 1 57 69

10.1370/afm.2908

36690490

21/1/57

PMC9870646

107

Schwartz

Moy

Rossetti

Elhadad

Cato

Clinician involvement in research on machine learning-based predictive clinical decision support for the hospital setting: A scoping review

J Am Med Inform Assoc 2021 03 01 28 3 653 663

10.1093/jamia/ocaa296

33325504

6039107

PMC7936403

108

Trinkley

Blakeslee

Matlock

Kao

Van Matre

Harrison

Larson

Kostman

Nelson

Lin

Malone

Clinician preferences for computerised clinical decision support for medications in primary care: a focus group study

BMJ Health Care Inform 2019 04 17 26 1

10.1136/bmjhci-2019-000015

31039120

bmjhci-2019-000015

PMC7062316

109

Sanz

Acha

García

M Ferrando

Co-design for people-centred care digital solutions: a literature review

Int J Integr Care 2021 04 30 21 2 16

10.5334/ijic.5573

33981193

PMC8086727

110

Brown

Hewner

The role of telehealth and clinical informatics in data driven primary care redesign

J Inform Nurs 2022 6 4 jin_21N4_A3

35733915

jin_21N4_A3

PMC9211055

111

Aarts

Callen

Coiera

Westbrook

Information technology in health care: socio-technical approaches

Int J Med Inform 2010 06

Information technology in health care

socio-technical approaches. Elsevier

389 90

112

Bate

Robert

Bringing User Experience to Health Care Improvement: The Concepts, Methods and Practices of Experience-Based Design 2007

Abingdon, Oxon, UK

Radcliffe Publishing

113

Cumpston

Chandler

Higgins

JPT

Thomas

Chandler

Cumpston

Page

Welch

Chapter II: Planning a Cochrane review

Cochrane Handbook for Systematic Reviews of Interventions version 6.5 2022

London, UK

Cochrane

114

Stoll

Izadi

Fowler

Green

Suls

Colditz

The value of a second reviewer for study selection in systematic reviews

Res Synth Methods 2019 12 18 10 4 539 545

10.1002/jrsm.1369

31272125

PMC6989049