The adoption of digital health platforms for collecting, sharing, and analyzing health information has shown positive associations with improvements in health care quality, service delivery, and clinical benefits including patient safety [1-4]. Electronic health records (eHRs) have become essential in acute care facilities as they enable the collection, sharing, and analysis of patient-related information, facilitating communication within and across health care settings. However, despite the substantial growth in the digitalization of health information exchange platforms, the complexity of many systems used by health care providers often poses challenges in achieving interoperability across different settings [5,6].

Differences in electronic systems used across different health settings can affect the exchange of relevant patient health and clinical information, especially during transitions of care or clinical handover [7,8]. Evidence indicates that the suboptimal communication between hospitals and external health care providers leads to discrepancies in medication records, duplication of tests, and avoidable delays in service provision, especially affecting vulnerable populations including those with low levels of health care literacy [9,10]. Hence, a coordinated health system with improved health information exchange, usability, and interoperability across health facilities and settings has significant potential to improve postacute care transition and overall patient safety [11].

Hospital discharge is a high-risk event where inaccurate or delayed transfer of clinical information, including medication plans, can significantly risk patient safety and cause medication-related issues [12,13]. Therefore, the clinical handover at the point of hospital discharge is a crucial step in patient care that determines the quality of care and patient safety. The introduction of electronic discharge summaries (eDS) has greatly improved the timely transmission of information to relevant stakeholders, mainly those in the primary care setting [8,14]. eDS, defined as “an end-to-end electronic transfer from the hospital to the community, using a secure messaging system, with the information populated using both pre-populated fields and manual transcription” [15] has seen increased adoption over the past decades. However, to further improve the quality of care and reduce communication delays between health settings, it is crucial that eDS should be user friendly [16]. This will help in minimizing avoidable patient harm incidents caused due to usability issues.

Usability is generally defined as “the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments” [17]. In the context of electronic systems, usability refers to whether the system is useful, usable, and satisfying for the intended users to enable completions of intended tasks in certain sequences [18]. Evidence indicates that there are several usability issues identified with eHRs, such as those related to data entry and alerts, interoperability issues, display, automation, and workflow [16]. These usability problems in addition to affecting the implementation of such systems have implications for patient safety such as medication error and use of inappropriate medication doses [16,19]. Evaluation of systems used to prepare eDS provides an opportunity to identify and improve usability issues with existing systems. Usability evaluation involves assessing performance, efficiency, and satisfaction of electronic interfaces and can identify usability issues with eHRs to thereby propose interventions to improve designs of interfaces, their learnability, and service efficiency [20]. While various international organizations have developed and provided guidelines on the content, form, and presentation of eDS [21-27], less is known about the usability of eDS and systems used to display eDS and their potential impact on quality of care.

Evidence from systematic reviews have identified a range of usability evaluation techniques applied broadly to eHRs, which include heuristic evaluation, cognitive walkthrough, think-aloud, user-testing, observation, coupled with use of questionnaires and interviews to assess participants’ perspectives and satisfaction [20,28]. However, there is limited evidence on the usability evaluations applied specifically to eDS. Therefore, the aim of this literature review was to identify the usability evaluation techniques that have been used to assess the usability of eDS.

This literature review is reported in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [29].

This review summarizes study findings on usability evaluation approaches used to assess eDS and eDS systems. The limited published evidence revealed the use of heterogenous usability evaluation techniques spanning from one conducted by experts to laboratory and user testing to the use of questionnaires and interviews. Broadly, our findings highlight that the use of heuristics (expert based) and think-aloud (user centered) were the most used methods for evaluation of eDS and eDS systems. Other techniques like eye-tracking, direct observation, questionnaire- and interview-based evaluations were also used in combination with either heuristic or think-aloud approaches.

Heuristic evaluation method, consistent with previous findings on eHRs [28], was used by most of the included studies for evaluation of eDS and eDS systems usability. This technique typically involves the application of a procedure including 3-5 experts to independently apply a set of best practices design (referred as heuristics) to identify usability flaws with system interfaces [44]. The heuristics used in evaluation are either defined a priori by experts or are derived from standard guidelines like the ergonomic criteria [46], which has 8 main domains around guidance, workload, explicit control, adaptability, error management, consistency, significance of codes, and compatibility.

Our review indicates that heuristic evaluation method can successfully identify a range of usability issues around readability, comprehensibility, organization, and content of eDS interfaces [38,39]. In addition to identifying usability flaws with user interfaces, heuristic evaluation also enabled assessment of the severity of usability problems. The severity of usability problems is often rated based on the 5-step severity scale developed by Nielsen et al [47], which is a tool widely applied to assess the usability of medical technologies and their impact on patient safety. This severity scale ranges in value from “0” for no usability problem to “4” for usability catastrophe, with mean scores of judgements from multiple evaluators used during heuristic usability evaluation [47].

While heuristic evaluation has the advantage of being more intuitive, efficient, and cheap, with less requirements for advanced planning and involvement of test users [44], it only identifies half of the usability problems that are related to the design of system interfaces [48]. A modified version called heuristic walkthroughs, which also involves the observation of end users, was associated with better detection of usability problems, mainly those characterized as moderate and severe [49]. This has been confirmed by one of the studies that reported heuristic walkthroughs to be effective in the identification of more severe usability problems [43]. Despite the advantages of heuristic evaluation, there are certain limitations associated with this approach. For example, the heterogenous nature of heuristics or guidelines applied in different settings indicates the lack of gold-standard guidelines applicable to every context [50]. Also, because heuristics are broadly defined, they are often interpreted and applied differently by different experts [50]. These limitations highlighted the need to explore alternative approaches of usability evaluation, preferably those that also consider input from end users.

The think-aloud technique is one of the controlled user testing approaches used by multiple studies in our review to evaluate the usability of eDS [33-35,37,43]. This method requires participants to verbalize their impressions about an interface while using it, enabling data collection from both direct observation and users’ self-reported statements [51,52]. This method has the advantage of providing insights into both design and learnability problems associated with systems [50]. Kernebeck et al [35] demonstrated that both effort and performance metrics can be effectively captured using a concurrent think-aloud evaluation approach, and emphasized the critical need to involve actual users from the start of the development process to enable a more transparent evaluation that meets the needs of end users. More importantly, the findings from Watbled et al [43] highlighted that this approach can be successfully integrated with heuristics, offering an advantage of a more holistic assessment of problems from both experts and users.

Eye-tracking is another controlled user testing method used for eDS usability evaluation [40]. In this method, eye-trackers record and analyze information on eye movement, fixation, and screen gaze to assess if the tasks involved are demanding. Questionnaires and semistructured interviews were also used to assess the usability issues with eHRs. Our review identified the use of the SUS [53]—a 10-item Likert tool that provides overall assessment of system usability. The SUS is a nonproprietary self-administered questionnaire with good validity and reliability; however, it is not robust and specific enough in identifying usability issues specific to eHRs.

The usability evaluation techniques used in the included studies, such as the use of heuristics, were not only used to identify issues related to eDS content, such as unnecessary or missing information, poor organization, and inconsistencies in formatting [32,38] but also used to understand the visualizations within eDS systems, including those associated with presenting medication lists and diagnoses [33,35,37,43]. The usability problems identified in the eDS systems had significant consequences, for example, the need for extended deadlines for task completion and errors in patient identification, which ultimately impacted the system's quality and performance [43]. These findings emphasize the importance of improving the speed and quality of systems when designing technologies for use in the context of eDS. It has been proposed that integrating usability testing methods during the development of these systems can potentially reduce adverse health events and outcomes.

In order to provide optimal and safe health services, eDS should provide clinically relevant, accurate, adequate, and clear display of relevant information. The content and quality of discharge summaries have implications for patient outcomes after their discharge from hospital [54]. While technological solutions can significantly improve the content and quality transfer of information, factors such as health literacy and individual patient differences are other important factors to consider during system implementation. This review highlights that this can be achieved through applying rigorous usability assessment techniques that require experts (heuristic evaluation and walkthroughs) approach with a user-based (think-aloud approach) method [50]. However, the limited number of studies assessing the usability of eDS or discharge instructions by patients with different levels of health literacy highlights the need for additional research.

Given the diverse user base of eHRs and discharge summaries in primary care settings, which includes physicians, nurses, pharmacists, and other health professionals, it is crucial to have systems that are easy to navigate, gather and select information, and interpret that information. Therefore, it is important to use a robust usability assessment approach that takes into account the wide range of users, including health professionals and patients, to develop a platform that can be used without significant challenges. Developing systems that can overcome usability issues, such as poor organization and display fragmentation, workflow interference, and cognitive overload, can affect the quality of information required to enable clinical decision making by health professionals and, therefore, continuity of care [55]. With emerging interest around International Patient Summary, which aims to provide a relatively generic means of communication for “unplanned, cross border care” [56], some of the identified usability techniques, especially those applicable to medication and condition summaries, can be used in this broader context.

Although most of the included studies assessed or explored different usability evaluation methods, the usability metrics used were heterogenous in nature. More studies focused on standardized usability metrics like efficiency, effectiveness, and satisfaction, as highlighted in the ISO (International Organization for Standardization) 9241-11 Ergonomics of human-system interaction [17] may shed light into the most effective approach for usability evaluation of eDS. Overall, usability evaluations applied on interfaces should aim to achieve adequate validity, thoroughness, and reliability [57]. In this context, considering the limitations with individual techniques, adopting a multimodal evaluation approach, for example, through combining heuristic evaluation with user testing methods or a questionnaire, may better achieve these objectives. More importantly, there should be an increased focus on developing and implementing usability evaluation techniques that consider factors such as learnability, regular use, error protection, accessibility, and maintainability, as highlighted in the ISO 9241-11 [17]. Another important consideration is the limited geographical locations covered by the included studies, which may limit the applicability of the findings in other settings with different electronic health systems and infrastructures. Lastly, the evidence concerning usability evaluation theories, approaches, and implementation frameworks specific to discharge summaries remains notably scarce. This highlights the need for further research in the area.

Even though we included a range of databases and gray literature sources, it is possible that we may have missed studies indexed in sources not included in this review. Our search strategy was specifically restricted to discharge summaries or instructions, which may have excluded usability evaluation techniques used in the context of EHRs in general. Some of these techniques identified in previous works focusing on EHRs could also be relevant to eDS [28]. We also acknowledge that despite our systematic and thorough approach, the potential for bias exists due to the reliance on a single reviewer for data extraction and quality appraisal.