This work was undertaken at a private not-for-profit teaching hospital, with approximately 150 beds, in Sydney, Australia. In 2015, the hospital implemented a single EMR in all wards and areas of the hospital. Before EMR implementation, a suite of fragmented clinical information systems was in place (an intensive care unit system, a general ward system, a medication management system, and a patient administration system). The single EMR was introduced to better support integration of care and to ensure efficient and effective delivery of health services. With respect to medication management, the system allowed prescribing, pharmacy review, and medication administration.

A core group of clinicians and administrative staff (information technology and billing) were consulted before EMR implementation, which included mainly heads of departments and units. Since EMR implementation, the organization elected to implement all medication-related alert types available in the system for prescribers: allergy and intolerance alerts, therapeutic duplication alerts, pregnancy alerts, and drug-drug interaction alerts. Not long after implementation, the alert configuration was changed so that only alerts classified as severe were displayed to doctors. All alerts were interruptive, appearing in an alert window that prevented prescribers from continuing with their orders until alerts were acknowledged. This could be done by selecting an override reason from a drop-down menu that appeared at the top of the alert screen (see Figure 1). The EMR included no medication administration alerts for nurses.

A Web-based survey was distributed to all doctors (n=1066) and nurses (n=411) at the site via hospital mailing lists (see Multimedia Appendix 1). Participation in the survey was voluntary, and no reimbursement was provided for participation. The survey was codeveloped with key stakeholders (researchers, doctors, nurses, pharmacists, and implementation staff) and comprised some basic demographic questions, questions on the current alerts in the system (for prescribers), and questions on preferences for changes to the system (for prescribers and nurses)—see Multimedia Appendices 2 and 3. Following distribution of the survey, a single reminder was sent out after 2 weeks. The survey was closed after 1 month.

Descriptive statistics were used for analysis of quantitative data. A general inductive approach was used for analysis of qualitative responses. Responses to free-text questions were reviewed independently by 2 researchers (WZ and BD) who came together to reach a consensus on key themes related to current alerts and redesign recommendations.

Ethics approval was obtained from the hospital’s human research ethics committee (reference number 5201833414318).

In total, 36 doctors and 60 nurses responded to the survey, representing a response rate of 3.37% (36/1066) and 14.5% (60/411), respectively (see Table 1).

Some respondents indicated that they were pleased the evaluation was being done:

However, on the whole, most doctors appeared highly dissatisfied with the EMR in general and made sarcastic or sweeping comments about the system in the free-text boxes of the survey. For example, a doctor said:

As shown in Table 2, approximately 90% (30/33) of doctors who responded held the view that too many alerts were triggered in the EMR, and only 9% (3/33) doctors indicated that the number of alerts was right.

When asked to indicate how frequently they read the alerts, responses were mixed, as shown in Table 3.

The free-text responses received in response to the question “I only read the alerts in the EMR when...” reflected this variability. Examples from doctors included:

When asked to rate each alert type on a Likert scale of usefulness, allergy and intolerances alerts were rated most positively. As shown in Table 4, these alerts were viewed to be sometimes useful by approximately half of the doctors in the survey. The other alert types were rated as never or rarely useful by the majority of doctors. When asked to indicate which alert type was the most useful, 60% (20/33) doctors selected allergy and intolerances alerts, and 24% (8/33) indicated that none of the alerts were useful.

Interestingly, when asked which of the alert types they would remove from the EMR (with more than one option possible), approximately half of the doctors (17/36, 47%) indicated that they would remove therapeutic duplication alerts, pregnancy alerts, and drug interaction alerts. In addition, 19% (7/36) also indicated that they would remove allergy and intolerance alerts.

Nurses did not receive medication administration alerts in the EMR; however, 97% (58/60) of the nurse respondents indicated that they should. When asked what types of alerts nurses should receive, most nurses selected all the alert types available to the doctors. Other suggested alert types included alerts for stat orders, overdue medications, blood thinning medications, dose alerts, and alerts warning when a nursing intervention was required (eg, digoxin).

When asked to indicate what alert severity level or levels should be included in the EMR, nurses and doctors expressed very different views. As shown in Table 5, nurses were more open to receiving alerts of all severities.

In response to a request for suggested changes to the alerts in the EMR, the most frequent response from doctors was to reduce alert numbers primarily by making the alerts more relevant. For example, doctors said:

Other common suggestions from doctor respondents were related to alert design. Current alerts were described as not user friendly, and doctors suggested using color coding to highlight the type or severity of alerts and suggested making alert content more concise:

This was consistent with what nurses viewed to be characteristic of a well-designed alert:

Some nurses also described what they believed to be the ideal content to be included in an alert screen. For example:

With respect to the current alerts in the EMR, prescribers reported that too many alerts were being triggered and that most alerts were not clinically relevant. As a consequence, alerts were not always being read, and many doctors reported experiencing alert fatigue. These findings confirm what had been suspected by the organization and are in line with many studies exploring user views of alerts [10,11]. Allergy alerts were perceived to be the most useful among the alert types, a finding also consistent with previous research [4,12]. With respect to recommended changes to the alerts, prescribers suggested that most alerts be removed and that alerts be more specific and less sensitive. Prescribers also recommended that alert interfaces be redesigned so that alert text is more concise, and color coding is used to signal alert type and severity of alerts. These suggestions were consistent with nurses’ preferences for alert content and display and reflect good human factors design [13,14].

Our survey highlighted some important lessons for the site and more broadly for other hospitals with plans to implement medication-related alerts in their EMR. Seeking user feedback on usefulness, volume, and design of alerts, even with this small-scale survey, proved to be extremely valuable; however, we suggest consulting users early and regularly. Our low response rate could in part reflect staff not checking emails or not having enough time to action emails. However, based on the free-text responses we received, it more likely reflected a workforce feeling disengaged and dissatisfied with the alerting system and associated implementation process. Early consultation would have allowed user views to be captured before negative mindsets were established and may have minimized any perception that the provision of feedback was futile. Continuous consultation with users is needed to ensure alert volume remains manageable and alert content remains relevant.

We recommend that organizations involve end users in the decision-making processes surrounding alert selection before EMR rollout as well as in implementation and ongoing evaluation. Involving users in EMR implementation more broadly has been shown to facilitate a sense of ownership of the system among users and to result in an EMR that aligns well with the needs of end users rather than the needs of the information technology department or leadership staff [15]. Consistent with previous research [16], our findings confirm that user engagement is particularly important for medication-related alerts. Engaging end users in the selection and implementation processes for alerts will also increase user awareness of the challenges associated with alert optimization (eg, vendor restrictions and complexity associated with identifying a set of high priority alerts).

One of the interesting findings that emerged from our survey was that in contrast to doctors, nurses showed a strong preference for enabling all alert types of all severities for nurses in the EMR. This could reflect their limited experience with alerts, as a common misconception held by clinicians who do not experience alerts themselves is the more alerts, the better. To allow end users to make informed decisions about alert selection, we recommend providing current users and prospective users with data on alert rates (eg, 50% of your orders/administrations/reviews will trigger an alert) to facilitate an understanding of the potential impact of alerts on their work and educating staff on the well-known risks associated with alert overload.

Our attempt to engage users in the redesign of medication-related alerts in an EMR was largely unsuccessful, with only a small number of doctors and nurses partaking in our survey. This likely reflects the delay between implementation and seeking feedback from end users. However, the feedback we received from survey respondents on usefulness, volume, and design of alerts proved to be extremely valuable. We recommend hospitals adopt a less-is-more approach and work closely with all end user groups before EMR implementation to determine the types, design, and content of alerts and implement these in their systems.