Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 08.10.19 in Vol 21, No 10 (2019): October

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/14683, first published May 13, 2019.

This paper is in the following e-collection/theme issue:

    Review

    Using Patient and Family Engagement Strategies to Improve Outcomes of Health Information Technology Initiatives: Scoping Review

    1Centre for Addiction and Mental Health, Toronto, ON, Canada

    2McMaster University, Hamilton, ON, Canada

    3University of Ottawa, Ottawa, ON, Canada

    4Western University, London, ON, Canada

    5Columbia University, New York, NY, United States

    6University of Victoria, Victoria, BC, Canada

    Corresponding Author:

    Gillian Strudwick, RN, PhD

    Centre for Addiction and Mental Health

    1001 Queen St W

    Toronto, ON, M6J1H4

    Canada

    Phone: 1 4165358501 ext 39333

    Email: gillian.strudwick@camh.ca


    ABSTRACT

    Background: Many health care organizations around the world have implemented health information technologies (ITs) to enhance health service efficiency, effectiveness, and safety. Studies have demonstrated that promising outcomes of health IT initiatives can be obtained when patients and family members participate and engage in the adoption, use, and evaluation of these technologies. Despite knowing this, there is a lack of health care organizations using patient and family engagement strategies to enhance the use and adoption of health ITs, specifically.

    Objective: This study aimed to answer the following three research questions (RQs): (1) what current frameworks or theories have been used to guide patient and family engagement in health IT adoption, use, implementation, selection, and evaluation?, (2) what studies have been done on patient and family engagement strategies in health IT adoption, use, implementation, selection, and evaluation?, and (3) what patient and family engagement frameworks, studies, or resources identified in the literature can be applied to health IT adoption, use, implementation, selection, and evaluation?

    Methods: This scoping review used a five-step framework developed by Arksey and O’Malley and adapted by Levac et al. These steps include the following: (1) identifying the RQ, (2) identifying relevant studies, (3) selecting studies, (4) charting relevant data, and (5) summarizing and reporting the result. Retrieved academic and grey literature records were evaluated using a literature review software based on inclusion and exclusion criteria by two independent reviewers. If consensus was not achieved, two reviewers would resolve conflicts by discussion. Research findings and strategies were extracted from the studies and summarized in data tables.

    Results: A total of 35 academic articles and 23 gray literature documents met the inclusion criteria. In total, 20 of the 35 included studies have been published since 2017. Frameworks found include the patient engagement framework developed by Healthcare Information and Management Systems Society and the patient and family engagement framework proposed by Carman et al. Effective strategies include providing patients with clear expectations and responsibilities and providing reimbursement for time and travel. The gray literature sources outlined key considerations for planning and supporting engagement initiatives such as providing patients with professional development opportunities, and embedding patients in existing governance structures.

    Conclusions: Several studies have reported their findings regarding successful strategies to engage patients and family members in health IT initiatives and the positive impact that can emerge when patients and family members are engaged in such initiatives in an effective manner. Currently, no framework has consolidated all of the key strategies and considerations that were found in this review to guide health care organizations when engaging patients and family members in a health IT–specific project or initiative. Further research to evaluate and validate the existing strategies would be of value.

    J Med Internet Res 2019;21(10):e14683

    doi:10.2196/14683

    KEYWORDS



    Introduction

    Globally, health care organizations have implemented health information technologies (ITs) to improve health service efficiency, effectiveness, and patient safety [1-9]. Health IT in this context refers to several technologies implemented in clinical settings to improve an aspect of clinical care. These technologies may include but are not limited to (1) electronic health records (EHRs; medical records in electronic form), (2) patient portals (Web access to a variety of functions such as viewing lab results and booking an appointment), (3) care coordination portals (electronic portals that support the coordination of care activities), and (4) mobile health apps (apps to track, learn, or monitor an aspect of one’s health) [10,11]. Some organizations have implemented patient portals that are tethered to hospital EHRs to provide patient access to their health information and provide patients with digital tools to take more control over their own health and care [12-14]. However, not all health care organizations have consistently taken full advantage of their health IT and systems [15]. Therefore, many of their potential benefits to patients, families, health professionals, and the health care system remain unseen [15].

    Several studies have shown promising results from health IT when engaging end users such as patients and family members in the adoption, use, and evaluation of a specific health IT such as an EHR [16,17]. One article revealed a decrease in medication errors at the hospital when patients were engaged in the implementation of an electronic medication administration system [16]. By engaging patients in interviews to determine the best methods for identifying mental health patients through the barcode scanning process, an improvement in medication scanning rates and reduction in medication errors was realized [7]. In another study, patients engaged in user interface testing of a patient portal improved the usability and uptake of the technology by marginalized patient populations [18]. These examples show that active ways (eg, incorporating patients on design teams and obtaining patient feedback preimplementation of a health IT) of engaging patients and family members in health IT activities can be beneficial [18]. If health care organizations begin to effectively engage their patients and families in the development, adoption, use, and evaluation of health IT more regularly, there may be an increasing number of benefits realized.

    Engaging patients and family members in decision making related to health services is not a new concept [19]. In fact, patients have engaged with health care organizations in many ways for several years now, such as advising health care institutions about health service delivery and patient-oriented health research [20]. In the United States, the Patient-Centered Outcomes Research Institute has advocated and funded the engagement of patients in many research initiatives [21]. However, patient engagement in health IT development, adoption, use, and evaluation is not as mature as it is in patient participatory medicine for health service delivery. The term, patient engagement, in this context broadly encompasses processes that involve informing and/or partnering with patients [22], to inform health service planning and delivery of the health IT. Engaging in a patient’s own health care is outside of the scope of this discussion.

    The objective of this scoping review was to explore studies that aim to improve outcomes of health IT initiatives through patient and family engagement and to explore practical strategies that health care organizations can leverage to meaningfully engage patients and families across the continuum of health IT initiatives. The review specifically focuses on how patients and family members can be involved in the health service planning and delivery context and not how they are more engaged in their own care.


    Methods

    This scoping review adopted the 5-step framework defined by Arksey and O’Malley [23] and updated by Levac et al [24]. These steps include the following: (1) identifying the research question (RQ), (2) identifying relevant studies, (3) selecting studies for data collection, (4) charting relevant data, and (5) summarizing and reporting the result [23,24]. Ethical approval was not required for this study. The following sections outline the steps followed to conduct the review.

    Step 1: Identifying the Research Questions

    The literature review was conducted to answer the following three RQs:

    • RQ1: What current frameworks, models, or theories have been used to guide patient and family engagement in health IT adoption, use, implementation, selection, and evaluation?
    • RQ2: What studies have been done on patient and family engagement strategies in health IT adoption, use, implementation, selection, and evaluation? And what are their results?
    • RQ3: What patient and family engagement frameworks (not specific to health IT), studies, and/or resources can be applied to health IT adoption, use, implementation, selection, and evaluation?

    The use of the 3 RQs was done to clearly articulate the scope of the review and explore literature that can be applied broadly to patient and family engagement in health IT projects or initiatives. The following assumptions were made to clarify the terminology used in the study when developing the RQs. The broader term health IT was used to represent various forms of IT used in health contexts such as (but not limited to) EHRs (patient records accessible via a computer) [25], patient portals (secure websites that allow patients to access their health record and other functions such as booking a medical appointment) [26], care coordination portals, and mobile health apps (use of mobile devices to provide health care services) [27]. RQs and terminology were refined through consultation with the members of the research team and a patient and family advisory committee. The patient and family advisory committee was based at a Canadian hospital located in Toronto, Ontario.

    Step 2: Identifying Relevant Studies

    With the guidance of a research librarian with experience conducting scoping reviews, a search strategy was developed using the following databases: MEDLINE, PsycINFO, Culmulative Index for Nursing and Allied Health Literature, Theses Canada, and the Education Resources Information Center. These databases are all commonly used in health science–related literature reviews with a focus on health services and were made available through the organizations in which the authors are employed. A primary search strategy was developed for the MEDLINE database (see Table 1) and adapted to be used for the other electronic databases.

    The search strategy used a combination of relevant Medical Subject Headings terms identified by the research librarian and keywords with Boolean terms (search combinations are shown in Table 1 via the use of the various search syntax shown above). The database search was supplemented with a search for unpublished gray literature related to patient engagement frameworks and toolkits. The search tool by the Canadian Agency for Drugs and Technologies in Health, Grey Matters, guided the gray literature search. A search using Google’s search engine was also completed. Additional methods for identifying relevant resources included the following: (1) reference searching from key article reference lists, (2) input from health IT experts, and (3) the patient and family advisor committee.

    The gray literature search used keywords to refine the search that included the following: ehealth, electronic medical records, electronic health records, patient portals, patient/families/caregiver engagement, ehealth adoption, use and evaluation, frameworks/ strategies/ resources/ tools/ toolkits, and health IT. Searches were completed between June and December 2018, and the screening process took part in early 2019.

    Inclusion/Exclusion Criteria

    Studies and frameworks in any clinical or health care setting were included. The studies and documents assessed were not limited to their date of publication or country of origin. Studies not published in English were excluded from this review. Systematic reviews were not eligible, but their reference lists were screened to find supplementary relevant studies. Studies that did not explicitly address patient or family engagement were excluded from the review. The focus of the review was on studies that involved, engaged, or empowered patients, families in the decision-making process, and across all stages of a health IT initiative for health service planning and delivery and not studies that focused on technologies that can be used to engage patients, families, and/or caregivers in their care. Studies that focused on patient engagement as an outcome of integrating and enabling technologies in the care process were, therefore, excluded.

    Table 1. Shows the search strategy developed for the MEDLINE database.
    View this table

    Step 3: Study Selection

    Numerous records were retrieved from the academic database search and gray literature search. Literature screening was completed using the Covidence systematic review software [28]. The software removed duplicates from the database search to aid with screening. Once duplicates were removed, 2 members of the research team independently screened the article titles and abstracts to determine if the full-text article should be retrieved and assessed. After the title and abstract screening, the inter-rater reliability score was calculated. Conflicts/discrepancies between screeners were resolved by discussion between the 2 screeners, and if consensus was not reached, a third member of the research team was consulted.

    Step 4: Charting the Data

    From each included study or gray literature document, pertinent information was extracted and summarized to address the RQs. This information included the following: descriptions of the study (study name, authorship, country of publication, journal published, study design, and population study), study methods (engagement strategies employed by researchers), and all study results related to the RQs (lessons learned and recommended engagement strategies).

    Step 5: Collating, Summarizing, and Reporting Results

    Results were collated, summarized, and reported based on the RQ or RQs that the article addressed. Descriptive statistics were used to summarize characteristics of the articles found through the database and gray literature search. A content analysis was performed on the identified studies to identify and record overarching themes that emerged. The approach to categorizing themes was an iterative, inductive process involving 2 members of the research team. The 2 members of the research team met after the data extraction process to create and refine the overarching themes identified. The recommended patient engagement strategies and considerations discussed in the included articles were also recorded.


    Results

    Characteristics of the Identified Studies

    A total of 1395 academic literature records, or gray literature documents, were retrieved from the academic and gray searches and expert consultation. During the abstract and title screening process, the inter-rater reliability between the 2 independent raters (KL/DM) was above 75% (Cohen kappa=.44). After the full-text records were assessed, 35 (n=35) academic articles and 23 (n=23) gray literature documents met the inclusion criteria (see Figure 1). The publication date of the academic articles and gray literature ranged between 2005 and 2018. The studies were conducted in the United States (n=14), and the remaining publications originated in other countries located in North America, Europe, and Asia. From the gray literature, 13 (n=13) documents were published in Canada, 4 (n=4) in Australia, 3 (n=3) in the United Kingdom, and 3 (n=3) in the United States. Table 2 provides an overview of the design of the academic and gray literature included in this review. Multimedia Appendix 1 provides the full data extraction table for all included records.

    Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses diagram indicating the flow of records reviewed during the literature search.
    View this figure
    Table 2. Design of the identified academic and gray literature (N=58).
    View this table

    RQ1: Current Health Information Technology–Related Frameworks, Models, or Theories

    A total of 2 articles outlined frameworks or models that have been used to effectively guide patient and family engagement in health IT adoption, use, implementation, selection, and evaluation [29,30]. In the first article that addresses RQ1, the Healthcare Information and Management Systems Society patient engagement framework was discussed. This framework was created to inform health care organizations on how to leverage health IT systems to implement patient engagement strategies [29]. The framework outlines 5 approaches to engagement that align with the International Association for Public Participation’s spectrum for community engagement and suggest tools such as patient-accessible records and patient-specific education to inform and partner with patients. Although these tools are not at the health service delivery level, the stages of engagement discussed in the framework are applicable to health IT health service planning and delivery. These stages of engagement are as follows: (1) inform, (2) consult, (3) involve, (4) collaborate, and (5) empower [22].

    The second article that addressed RQ1 is a study by Walker et al that outlined an evaluative model for health IT implementation [30]. Walker et al engaged patient groups through a series of focus groups and an online forum to understand the evaluation needs of patients related to an inpatient portal implementation. Through a nominal group approach, patients specified the following 2 categories of patient-specific outcomes that should be incorporated in an evaluation of an inpatient portal: (1) outcomes related to the explicit use of the inpatient portal and (2) outcomes related to the tacit knowledge gained by having access to an inpatient portal [30]. Explicit use of the inpatient portal may result in a change in patient satisfaction, which can be a measure of patient outcomes as a result of portal implementation and use [30]. Tacit knowledge that is gained from using the inpatient portal could result in change in patient engagement in health IT, which can be a measurable outcome [30]. The model demonstrates that soliciting patient feedback on explicit portal use and changes in tacit knowledge is needed for a multifaceted evaluation of an inpatient portal implementation.

    RQ2: Patient and Family Engagement Strategies in Health Information Technology Initiatives

    A total of 19 (n=19) studies were identified that utilized patient and family engagement strategies in health IT adoption, use, implementation, selection, and evaluation. A finding of 2 studies related to the use of engaging patients through multiple digital modalities. A study by Athilingam et al engaged patients on the design, potential features, and the ease of use of a mobile health app to improve self-care in heart failure [31]. Interviews were conducted with 10 patients and 4 cardiologists before the initial prototype was developed. The authors concluded that existing mobile health apps have not been widely adopted and suggested the mixed use of internet, email, and text messages for promoting better communication and long-term engagement with digital health apps [31]. Another study by Lafata et al reiterated the need for different digital modalities to effectively engage patients from vulnerable patient populations [32].

    Multidisciplinary, Team-Based Approach

    A total of 4 (n=4) studies suggested that a multidisciplinary, team-based approach would be an effective engagement strategy for engaging patients and families in the use of health IT [33-36]. A study by Ackerman et al examined the patient engagement strategies during the implementation of a patient portal at 5 community health centers [33]. Portal champions reminded clinicians and staff to encourage patients to sign up for the portal. Volunteers, front desk clerks, and medical assistants provided enrollment assistance to patients and used clinic computers to demonstrate to patients how to use portal services. The study by Ackerman et al found that the uptake and use of the portal improved when patients were engaged by trusted staff members or clinicians. Results from the study by Raval et al concluded that the engagement of a pediatric surgeon and physician assistant was crucial to the success of recruiting and engaging patients in the development of a mobile app for colorectal disease management [34]. As the clinical team was available during the planning stages of the research project, it helped ease the process of engaging patients in the study. As a result, pediatric patients and their family members who were already visiting the hospital were successfully recruited and provided useful feedback. Similar recommendations were made by Krist et al and Shapiro-Mathews et al [35,36]. In a study by Krist et al, patient engagement strategies in primary care practices were evaluated [35]. Primary care offices had registration staff pass out information to patients, had nurses discuss how to sign up for a patient portal, and other clinicians communicate the value of signing up for the patient portal. Krist et al concluded that a team-based approach to engaging patients positively influenced the uptake of the patient portal compared with the clinician-dependent approach to engaging patients. The article by Shapiro-Mathews et al outlined an institutional strategy for mobile health technologies that requires an interdisciplinary approach [36]. A clinical nurse specialist and other health care professionals can facilitate a team-based approach to engage patients, provide patient education, and inform the design of mobile apps that meet the needs of patients.

    Training/Education of Patients

    A total of 8 (n=8) articles highlighted the importance of training/education of patients in the success of health IT adoption, use, implementation, selection, and evaluation. The results from a study by Anshari et al showed that the availability of a Web-based health educator is important to improving the health literacy of patients and empowering patients to control their own health and health information [37]. Greysen et al conducted a randomized controlled trial where patients in the intervention arm received tailored, structured education regarding the use of a patient portal at the bedside [38]. Study results suggested that bedside portal training produced a trend of increased ability to log in and navigate the portal, satisfaction with portal use, and frequency of portal use after discharge. Van den Bulck et al explored patient needs, expectations, and attitudes toward a patient portal by administering an online survey to recruited patients [39]. Results showed that digital health literacy is a key factor to portal adoption and providing training to patients could provide exposure to using the portal and create appropriate expectations of what the portal is capable of. Another study by Wildenbos et al explored experiences and perspectives of older adult patients using a patient portal 1-year postimplementation of the technology [40]. The results from the study by Wildenbos et al concluded that health literacy level of patients was a strong factor that influenced the patient’s overall interest and perceived ability to use the patient portal. Although the previously discussed training/education methods may be most directed at engaging patients in their own care, similar methods could be used to orient patients before engaging in a health IT initiative aimed at supporting adoption, use, implementation, selection, and evaluation.

    Training of Health Care Providers

    In addition to training patients on the use of health IT, the study by Raval et al also recommended that health care providers may need to be trained to address potentially low levels of self-efficacy of employing effective patient engagement methods [34]. Wildenbos et al also emphasized the need for health care providers to be trained regarding how to effectively engage patients [41]. Likewise, a study by Metting et al explored patient needs and opinions through focus groups to facilitate the development of patient Web portals [42]. A training activity that was recommended included training health care providers to effectively communicate with patients with regards to engagement specific activities.

    RQ3: General Patient and Family Engagement Frameworks, Studies, and Resources

    A total of 13 (n=13) academic sources and 23 (n=23) gray literature sources were found that employed patient and family engagement strategies for research and clinically relevant projects that were not health IT–specific, but the principles or findings embedded within them could potentially be applied to this context. In turn, relevant principles and findings may be included in a future health IT–specific patient and family engagement framework or set of recommendations for health care organizations to utilize. Of the 13 academic articles, 6 (n=6) explicitly outlined frameworks that can be used to involve patients and family members within nonhealth IT contexts such as research, health service delivery, and quality improvement.

    A framework proposed by Carman et al was widely used to inform effective patient and family engagement strategies in health IT adoption, implementation, use, selection, and evaluation [43] (see Figure 2). The framework outlines 3 categories of engagement activities across a continuum: (1) consultation, (2) partnership, and (3) shared leadership. These 3 categories can be applied in different levels of the health care system, which was segmented by the authors as follows: individual care, organization governance, and government policy. The framework highlighted individual factors that can potentially impact a patient’s willingness and ability to engage with the health care system such as health literacy and education level. At the institutional level, health care organizations and staff can also encourage patient engagement through demonstrating that patient participation and leadership is imperative to the achievement of organizational goals [43].

    Policies and practices that can influence patient engagement can create expectations that patients can and will serve as advisors and decision makers on committees and patient-centered councils. Bridgepoint Hospital in Toronto, Canada, implemented practices that brought patients, families, and hospital staff together to redesign health care services at the hospital and improve the overall patient experience [44]. Patients and family members at the hospital were recruited as advisors to review the hospital quality improvement processes.

    At the government level, policy makers can create mechanisms so that patients can provide input in public policy, such as public deliberation sessions, town hall meetings, public hearings, or regulatory comment processes [43]. Nonprofit organizations and government agencies can also aid in creating funding mechanisms requiring patient participation in the decision-making process. These efforts to encourage patients and family member participation in quality improvement processes broadly can be applied specifically to the health IT context, creating mechanisms that encourage patients and families to provide their input in health IT–related projects and policies.

    Figure 2. Multidimensional framework for patient and family engagement in health and health care (permission from author and publisher obtained to use image, and copyright obtained).
    View this figure

    A number of the identified studies that addressed RQ3 utilized different strategies to engage patients in a clinical research network or to engage the youth in projects led by academic researchers (see Table 3).

    Guidance documents and reports identified in the gray literature highlighted key attributes to patient engagement broadly in the health care and research context. Gray literature sources highlighted key considerations to guide health care organizations in preparing for engagement activities (see Table 4).

    Table 3. A list of common strategies used in identified academic studies to engage patients and family members.
    View this table
    Table 4. Key considerations from existing engagement frameworks and toolkits when preparing for engagement.
    View this table

    Additional considerations and characteristics were highlighted to support patients, family members, and staff in health care organizations during engagement activities (see Table 5). In addition to preparing and supporting engagement, 9 (n=9) gray literature sources highlighted the importance of conducting an evaluation to ensure that initiatives to engage patients and family members are meaningful and provide value for all stakeholders [21,57,59,61-65,67]. Studies encouraged the use of quantitative measures and measuring against specified objectives to demonstrate the value of engagement [57,67]. Different evaluative methods were described such as providing evaluation forms, conducting surveys, and engaging in discussions with patients to provide the opportunity for giving constructive feedback. The use of surveys and feedback forms allows for organizations to solicit feedback anonymously, which may be preferred by some patients and family members [62,63,69].

    Table 5. Key considerations to support patients and family members during engagement activities outlined by gray literature sources.
    View this table

    Discussion

    Primary Findings

    Patient and family engagement has become a topic of increasing interest in the research community in the last several years. In fact, the majority (57%, 33/58) of the included studies in this review have been published since 2017. Identified studies have highlighted the importance of involving and partnering with patients in health IT design and development [51,73]. Several studies have used a variety of strategies and outlined considerations when engaging patients and family members in health IT–related projects. These studies outlined several considerations for engaging patients and families, mainly when preparing for engagement or in the early stages of the engagement process (see Table 4). Strategies that were consistent among the identified studies include providing incentives and reimbursement for patients, clarification of patient roles and responsibilities, and demonstrating the value of engagement to patients. However, there is a limited amount of literature that have established a framework, model, or theory, or set of recommendations that can be used to guide patient and family engagement in health IT initiatives. As a result, there is no standardized methodology or resource for engaging patients and family members effectively for health IT initiatives. The development of a practical framework (borrowing concepts, recommendations, and relevant principles from the nonhealth IT literature) to guide health care organizations and health care providers within this unique context is needed. One main limitation for the development of a standardized framework is that it may not capture the unique needs of all diverse patient groups. Therefore, additional resources may be required to supplement and guide effective patient and family engagement for health IT initiatives; however, starting with a resource that health care organizations can use to engage in this important work, is needed.

    Patient engagement is a broad term that can be defined by varying levels of involvement from the patient. The overall goal of patient engagement may not always be about moving to a higher level of engagement and patient empowerment. The study by Hamilton et al outlined a key recommendation of patients valuing the freedom to become gradually more engaged at their own pace. Therefore, desired objectives and the impact of engagement need to be discussed and agreed upon between the organization and the engaged stakeholders. Many health care organizations recognize the varying degrees of engagement. A total of 9 (n=9) citations found in this review referred and/or adapted the participation spectrum outlined by the International Association for Public Participation that identifies different levels of stakeholder and community participation [22]. These findings are relevant not only in the broader health care or health research settings but can be applied to health IT contexts.

    Comparison With Past Work

    This scoping review adds to existing literature reviews that have articulated successful engagement characteristics and approaches within the health research context [74]. The impact of engagement can be profound. The review by Manafo et al documented outcomes when engaging patients in health research that include the following: (1) patients feeling empowered, (2) improved trust between researcher and patients, and (3) decreased attrition of study participants. Increased engagement has also positively impacted health IT projects in different ways such as increased usage of health IT, satisfaction with health IT use [35,37,38], and the obtainment of quality/safety-related outcomes [17]. A review on patient involvement in health research by Shippee et al concluded that available literature on patient involvement focused on 1 research phase and particularly earlier stages of research. Similarly, academic studies have involved patients in specific stages of health IT projects, such as the usability testing or the implementation of an inpatient portal and not necessarily throughout all stages of the health IT life cycle. An article by Petersen articulated that despite patients being actively involved in conducting research, patients are often not involved with setting the research agenda, evaluating results, or discussing next steps [20]. Similarly, there is limited evidence reporting or evaluating engagement strategies across multiple phases of a health IT project. As a result, there is a need for further research on sustained engagement throughout different stages of health IT projects and how strategies may differ depending on whether the organization is in a development or implementation phase of a health IT project. This is where identifying findings from the nonhealth IT literature may add value.

    Studies have highlighted the important role of health providers in the engagement and activation of patients [57,65]. A study by Graffigna concluded that health providers have a crucial role in influencing the engagement and activation of type 2 diabetic patients in using health IT tools to manage their health condition [75]. Furthermore, several academic and gray literature sources have highlighted the need to incorporate patients and families in underserved populations in health IT initiatives. Involvement and feedback that reflects the diversity of the community can allow organizations to gain a better understanding of the diversity in patient needs [59,68,76,77]. Studies suggested that there were economic and ethnic disparities associated with the use and adoption of health IT [78,79]. Future studies could explore methods in which current engagement strategies can be adapted to effectively engage patients and families from underserved populations. There is also opportunity for health care organizations that primarily work with underserved populations to document and disseminate their strategies in effectively engaging these populations.

    Gray literature sources have outlined the importance of evaluating engagement activities to quantify the value of engagement and provide constructive feedback on how engagement initiatives can be improved [63,67]. Evaluation methods that have been recommended include having feedback discussions with patients, providing evaluation forms, and surveys. Other sources have also considered standardizing evaluation by building evaluation components into project plans and leveraging existing standardized tools to evaluate the process and impact of engagement [56,59]. A study by Abelson et al developed a patient engagement evaluation tool that can be used by health care organizations broadly. The evaluation tool consists of 3 unique questionnaires used to evaluate the following 4 evaluation principles: (1) integrity of design and process, (2) influence and impact, (3) participatory culture, (4) collaboration, and (5) common purpose [80].

    Despite the number of tools that have been developed to evaluate public and patient engagement, there is a lack of consistency on how engagement strategies are evaluated, and few studies have quantitatively evaluated measures of engagement. A literature review by Esmail et al outlined that there were only 2 studies that evaluated patient engagement within the health research context using quantitative measures [81]. A review by Dukhanin et al identified other existing evaluation tools but concluded that the methodology of each tool varies significantly [82]. Therefore, implications for research include the validation of existing evaluation tools and the combined use of qualitative and quantitative tools to assess engagement. Furthermore, many existing evaluation tools rely heavily on process metrics and measure perceived benefits of engagement [82]. There is a need for health care organizations to increase adoption and use of outcome metrics, such as changes in patient knowledge and attitudes toward engagement, to evaluate patient, public, and community engagement. Meaningful evaluation efforts require capacity and commitment from health care organizations [80]. An implication for health care organizations is to internally build the capacity and culture that supports evaluation efforts to improve the engagement process for patients, families, and organizations.

    Limitations

    This review has a few limitations that should be considered when reviewing its findings. The exploration of engagement strategies broadly has led to several considerations and recommendations for how to engage patients and families in health IT initiatives. As identified in the literature, there is a spectrum that exists within the context of patient engagement. Patients can be actively or passively involved in health IT initiatives, and this distinction should be further explored throughout the stages of engagement within the health IT context. A more focused approach that applies to specific health care settings may be completed to solicit results that are appropriate for specific contexts.

    Regarding the review methodology, there were challenges with the search strategy regarding the use of broad terms such as theory, model, and frameworks. The terms theory, model, and frameworks were taken from common terminology and approaches used in implementation science [83]. This was done to capture as many different types of engagement tools that could guide future engagement strategies; but as a result, there were a significant number of studies that did not meet the appropriate inclusion criteria for this review but were identified in the initial search. The review included studies published in English only. There is a possibility that relevant studies and resources from health care organizations in non-English speaking countries that have studied or developed engagement strategies in health IT settings were excluded.

    Conclusions

    Several studies and gray literature documents identified in this review have reported their findings on successful strategies to engage patients and family members in health care and the positive impact that can emerge when patients and family members are engaged. Several studies have employed a variety of engagement strategies to engage diverse patient populations in health IT projects. Currently, no framework, set of recommendations, or resource document has consolidated all of the lessons learned and considerations to guide health care organizations when engaging patients and family members in a health IT–specific project. There is much to learn and incorporate from the nonhealth IT–specific work that has already been done. With the increasing number of studies reporting their findings of engaging patients in health IT–related initiatives, continuing efforts to evaluate and validate these engagement strategies is needed.

    Acknowledgments

    The authors would like to acknowledge the patient and family advisory committee for their contributions in conducting this scoping review. The authors would also like to thank the research librarian who provided guidance for the search strategy utilized. Finally, the authors would like to thank the Canadian Institutes of Health Research for funding this study.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Academic literature data extraction table.

    PDF File (Adobe PDF File)205 KB

    References

    1. Better Information for Improved Health: A Vision for Health System Use of Data in Canada. Ottawa, ON: Canadian Institute for Health Information; 2013.
    2. Chang F, Gupta N. Progress in electronic medical record adoption in Canada. Can Fam Physician 2015 Dec;61(12):1076-1084 [FREE Full text] [Medline]
    3. Powell KR, Myers CR. Electronic patient portals: patient and provider perceptions. Online J Nurs Informatics 2018;22(1):1 [FREE Full text]
    4. Schoen C, Osborn R, Squires D, Doty M, Rasmussen P, Pierson R, et al. A survey of primary care doctors in ten countries shows progress in use of health information technology, less in other areas. Health Aff (Millwood) 2012 Dec;31(12):2805-2816. [CrossRef] [Medline]
    5. Institute for e-Health Policy. 2014. A Glimpse at EHR Implementation Around the World: The Lessons the US Can Learn   URL: https:/​/www.​e-healthpolicy.org/​sites/​e-healthpolicy.org/​files/​A_Glimpse_at_EHR_Implementation_Around_the_World1_ChrisStone.​pdf [accessed 2019-05-08]
    6. Gray BH, Bowden T, Johansen I, Koch S. Electronic health records: an international perspective on 'meaningful use'. Issue Brief (Commonw Fund) 2011 Nov;28:1-18. [Medline]
    7. Sulkers H, Tajirian T, Paterson J, Mucuceanu D, MacArthur T, Strauss J, et al. Improving inpatient mental health medication safety through the process of obtaining HIMSS Stage 7: a case report. JAMIA Open 2019;2(1):35-39. [CrossRef]
    8. Kipping S, Stuckey MI, Hernandez A, Nguyen T, Riahi S. A web-based patient portal for mental health care: benefits evaluation. J Med Internet Res 2016 Nov 16;18(11):e294 [FREE Full text] [CrossRef] [Medline]
    9. FitzHenry F, Wells N, Doran J, Hughart K, Levy M, Doulis J. Applying bar code medication administration to make a difference in adverse drug events with potential for harm: lessons learned. Comput Inform Nurs 2013 Oct;31(10):457-462. [CrossRef] [Medline]
    10. OpenNotes – Patients and Clinicians on the Same Page. 2018.   URL: https://www.opennotes.org/ [accessed 2018-02-08]
    11. Patel V, Johnson C. HealthIT. 2018. Individuals’ Use of Online Medical Records and Technology for Health Needs   URL: https://www.healthit.gov/sites/default/files/page/2018-03/HINTS-2017-Consumer-Data-Brief-3.21.18.pdf [accessed 2019-09-04]
    12. HealthIT. 2017. What is a Patient Portal?   URL: https://www.healthit.gov/providers-professionals/faqs/what-patient-portal [accessed 2019-09-05]
    13. Irizarry T, DeVito DA, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015 Jun 23;17(6):e148 [FREE Full text] [CrossRef] [Medline]
    14. Irizarry T, Shoemake J, Nilsen ML, Czaja S, Beach S, DeVito Dabbs DA. Patient portals as a tool for health care engagement: a mixed-method study of older adults with varying levels of health literacy and prior patient portal use. J Med Internet Res 2017 Mar 30;19(3):e99 [FREE Full text] [CrossRef] [Medline]
    15. Government of Canada Publications. 2010. Electronic Health Records in Canada: An Overview of Federal and Provincial Audit Reports   URL: http://publications.gc.ca/site/eng/367717/publication.html [accessed 2019-09-05]
    16. Paoletti RD, Suess TM, Lesko MG, Feroli AA, Kennel JA, Mahler JM, et al. Using bar-code technology and medication observation methodology for safer medication administration. Am J Health Syst Pharm 2007 Mar 1;64(5):536-543. [CrossRef] [Medline]
    17. Strudwick G, Clark C, McBride B, Sakal M, Kalia K. Thank you for asking: exploring patient perceptions of barcode medication administration identification practices in inpatient mental health settings. Int J Med Inform 2017 Sep;105:31-37. [CrossRef] [Medline]
    18. Eyasu T, Leung K, Strudwick G. Guiding improvements in user experience: results of a mental health patient portal user interface assessment. Stud Health Technol Inform 2019;257:110-114. [Medline]
    19. Perfetto EM, Harris J, Mullins CD, dos Reis S. Emerging good practices for transforming value assessment: patients' voices, patients' values. Value Health 2018 Apr;21(4):386-393 [FREE Full text] [CrossRef] [Medline]
    20. Petersen C. Patient informaticians: turning patient voice into patient action. JAMIA Open 2018;1(2):130-135. [CrossRef]
    21. Patient-Centered Outcomes Research Institute (PCORI). 2019. Improving outcomes important to patients   URL: https://www.pcori.org/ [accessed 2019-07-13]
    22. IAP2 Canada. 2019. IAP2 Spectrum   URL: https://iap2canada.ca/Resources/Documents/0702-Foundations-Spectrum-MW-rev2%20(1).pdf [accessed 2019-07-13]
    23. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005 Feb;8(1):19-32 [FREE Full text] [CrossRef]
    24. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010 Sep 20;5:69 [FREE Full text] [CrossRef] [Medline]
    25. Häyrinen K, Saranto K, Nykänen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008 May;77(5):291-304. [CrossRef] [Medline]
    26. Canada Health Infoway: Digital Health in Canada. 2019. Electronic Health Records   URL: https://www.infoway-inforoute.ca/en/solutions/digital-health-foundation/electronic-health-records [accessed 2019-07-13]
    27. Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al. The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 2013;10(1):e1001362 [FREE Full text] [CrossRef] [Medline]
    28. Veritas Health Innovation. Covidence - Better Systematic Review Management. 2016.   URL: https://www.covidence.org/home [accessed 2019-09-05]
    29. Healthcare Information and Management Systems Society. 2014. HIMSS Patient Engagement Framework.   URL: https://www.himss.org/himss-patient-engagement-framework [accessed 2019-07-13]
    30. Walker DM, Hefner JL, Sieck CJ, Huerta TR, McAlearney AS. Framework for evaluating and implementing inpatient portals: a multi-stakeholder perspective. J Med Syst 2018 Jul 16;42(9):158. [CrossRef] [Medline]
    31. Athilingam P, Clochesy JM, Labrador MA. Intervention mapping approach in the design of an interactive mobile health application to improve self-care in heart failure. Comput Inform Nurs 2018 Feb;36(2):90-97. [CrossRef] [Medline]
    32. Lafata JE, Miller CA, Shires DA, Dyer K, Ratliff SM, Schreiber M. Patients' adoption of and feature access within electronic patient portals. Am J Manag Care 2018 Nov 1;24(11):e352-e357 [FREE Full text] [Medline]
    33. Ackerman SL, Sarkar U, Tieu L, Handley MA, Schillinger D, Hahn K, et al. Meaningful use in the safety net: a rapid ethnography of patient portal implementation at five community health centers in California. J Am Med Inform Assoc 2017 Sep 1;24(5):903-912 [FREE Full text] [CrossRef] [Medline]
    34. Raval MV, Taylor N, Piper K, Thakore M, Hoff K, Owens S, et al. Pediatric patient and caregiver preferences in the development of a mobile health application for management of surgical colorectal conditions. J Med Syst 2017 Jul;41(7):105. [CrossRef] [Medline]
    35. Krist AH, Peele E, Woolf SH, Rothemich SF, Loomis JF, Longo DR, et al. Designing a patient-centered personal health record to promote preventive care. BMC Med Inform Decis Mak 2011 Nov 24;11:73 [FREE Full text] [CrossRef] [Medline]
    36. Shapiro-Mathews E, Barton AJ. Using the patient engagement framework to develop an institutional mobile health strategy. Clin Nurse Spec 2013;27(5):221-223. [CrossRef] [Medline]
    37. Anshari M, Almunawar M, Low PK, Wint Z, Younis MZ. Adopting customers' empowerment and social networks to encourage participations in e-health services. J Health Care Finance 2013;40(2):17-41. [Medline]
    38. Greysen SR, Harrison JD, Rareshide C, Magan Y, Seghal N, Rosenthal J, et al. A randomized controlled trial to improve engagement of hospitalized patients with their patient portals. J Am Med Inform Assoc 2018 Dec 1;25(12):1626-1633. [CrossRef] [Medline]
    39. van den Bulck SA, Hermens R, Slegers K, Vandenberghe B, Goderis G, Vankrunkelsven P. Designing a patient portal for patient-centered care: cross-sectional survey. J Med Internet Res 2018 Oct 1;20(10):e269 [FREE Full text] [CrossRef] [Medline]
    40. Wildenbos GA, Maasri K, Jaspers M, Peute L. Older adults using a patient portal: registration and experiences, one year after implementation. Digit Health 2018;4:2055207618797883 [FREE Full text] [CrossRef] [Medline]
    41. Wildenbos GA, Horenberg F, Jaspers M, Peute L, Sent D. How do patients value and prioritize patient portal functionalities and usage factors? A conjoint analysis study with chronically ill patients. BMC Med Inform Decis Mak 2018 Nov 21;18(1):108 [FREE Full text] [CrossRef] [Medline]
    42. Metting E, Schrage AJ, Kocks JW, Sanderman R, van der Molen T. Assessing the needs and perspectives of patients with asthma and chronic obstructive pulmonary disease on patient web portals: focus group study. JMIR Form Res 2018 Nov 22;2(2):e22 [FREE Full text] [CrossRef] [Medline]
    43. Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood) 2013 Feb;32(2):223-231. [CrossRef] [Medline]
    44. Health Quality Ontario. 2016. Engaging with Patients: Stories and Successes from the 2015/2016 Quality Improvement Plans.   URL: http://www.hqontario.ca/Portals/0/documents/qi/qip/engaging-with-patients-en.pdf [accessed 2019-07-19]
    45. Arkind J, Likumahuwa-Ackman S, Warren N, Dickerson K, Robbins L, Norman K, et al. Lessons learned from developing a patient engagement panel: an OCHIN report. J Am Board Fam Med 2015;28(5):632-638 [FREE Full text] [CrossRef] [Medline]
    46. Boyer AP, Fair AM, Joosten YA, Dolor RJ, Williams NA, Sherden L, et al. A multilevel approach to stakeholder engagement in the formulation of a clinical data research network. Med Care 2018 Oct;56(Suppl 10 Suppl 1):S22-S26 [FREE Full text] [CrossRef] [Medline]
    47. Hawke LD, Relihan J, Miller J, McCann E, Rong J, Darnay K, et al. Engaging youth in research planning, design and execution: practical recommendations for researchers. Health Expect 2018 Dec;21(6):944-949 [FREE Full text] [CrossRef] [Medline]
    48. Hamilton CB, Hoens AM, Backman CL, McKinnon AM, McQuitty S, English K, et al. An empirically based conceptual framework for fostering meaningful patient engagement in research. Health Expect 2018 Feb;21(1):396-406 [FREE Full text] [CrossRef] [Medline]
    49. Shelef DQ, Rand C, Streisand R, Horn IB, Yadav K, Stewart L, et al. Using stakeholder engagement to develop a patient-centered pediatric asthma intervention. J Allergy Clin Immunol 2016 Dec;138(6):1512-1517. [CrossRef] [Medline]
    50. Warren NT, Gaudino JA, Likumahuwa-Ackman S, Dickerson K, Robbins L, Norman K, et al. Building meaningful patient engagement in research: case study from ADVANCE clinical data research network. Med Care 2018 Oct;56(Suppl 10 Suppl 1):S58-S63 [FREE Full text] [CrossRef] [Medline]
    51. Chung AE, Vu MB, Myers K, Burris J, Kappelman MD. Crohn's and Colitis foundation of America partners patient-powered research network: patient perspectives on facilitators and barriers to building an impactful patient-powered research network. Med Care 2018 Oct;56(Suppl 10 Suppl 1):S33-S40 [FREE Full text] [CrossRef] [Medline]
    52. Faulkner M, Alikhaani J, Brown L, Cruz H, Davidson D, Gregoire K, et al. Exploring meaningful patient engagement in ADAPTABLE (aspirin dosing: a patient-centric trial assessing benefits and long-term effectiveness). Med Care 2018 Oct;56(Suppl 10 Suppl 1):S11-S15 [FREE Full text] [CrossRef] [Medline]
    53. Coathup V, Teare HJ, Minari J, Yoshizawa G, Kaye J, Takahashi MP, et al. Using digital technologies to engage with medical research: views of myotonic dystrophy patients in Japan. BMC Med Ethics 2016 Aug 24;17(1):51 [FREE Full text] [CrossRef] [Medline]
    54. Suarez-Balcazar Y. Empowerment and participatory evaluation of a community health intervention: implications for occupational therapy. OTJR (Thorofare N J) 2016 Jun 21;25(4):133-142. [CrossRef]
    55. Canadian Institutes of Health Research. 2014. Strategy for Patient-Oriented Research - Patient Engagement Framework.   URL: http://www.cihr-irsc.gc.ca/e/48413.html [accessed 2019-07-14]
    56. Health Quality Ontario. 2016. Engaging With Patients and Caregivers About Quality Improvement: A Guide for Health Care Providers.   URL: http://www.hqontario.ca/portals/0/documents/qi/qip/patient-engagement-guide-1611-en.pdf [accessed 2019-07-13]
    57. Government of South Australia: SA Health. 2013. Guideline: Guide for Engaging with Consumers and the Community.   URL: https:/​/www.​sahealth.sa.gov.au/​wps/​wcm/​connect/​f8d1d0004e454788aa0caf8ba24f3db9/​Guideline_Guide+for+Engaging+with+Consumers+and+Community_Oct2015.​pdf?MOD=AJPERES&CACHEID=f8d1d0004e454788aa0caf8ba24f3db9 [accessed 2019-07-14]
    58. Hailey D. Institute of Health Economics. 2005. Consumer Involvement in Health Technology Assessment.   URL: https://www.ihe.ca/download/consumer_involvement_in_health_technology_assessment.pdf [accessed 2019-07-14]
    59. Health Consumers Queensland. 2017. Consumer and Community Engagement Framework.   URL: http://www.hcq.org.au/our-work/framework/ [accessed 2019-07-14]
    60. Hayes H, Buckland S, Tarpet M. INVOLVE. 2012. Briefing Notes For Researchers: Public Involvement in NHS, Public Health and Social Care Research.   URL: http://www.invo.org.uk/wp-content/uploads/2014/11/9938_INVOLVE_Briefing_Notes_WEB.pdf [accessed 2019-07-14]
    61. Health Quality Ontario. 2017. Ontario’s Patient Engagement Framework: Creating a Strong Culture of Patient Engagement to Support High Quality Health Care.   URL: https://www.hqontario.ca/Portals/0/documents/pe/ontario-patient-engagement-framework-en.pdf [accessed 2019-07-14]
    62. Paediatric Integrated Cancer Services. 2013. A Toolkit for Consumer Participation and Engagement.   URL: https://pics.org.au/health-professionals/engaging-consumers/ [accessed 2019-07-14]
    63. James Lind Alliance. 2018. James Lind Alliance Guidebook.   URL: http://www.jla.nihr.ac.uk/jla-guidebook/ [accessed 2019-07-14]
    64. Vancouver Coastal Health. 2017. Community Engagement Framework.   URL: http://www.vch.ca/Documents/CE-Framework.pdf [accessed 2019-07-14]
    65. Center for Advancing Health. American Hospital Association. 2014. A New Definition of Patient Engagement: What is Engagement and Why is it Important?   URL: http:/​/www.​hret-hiin.org/​resources/​display/​a-new-definition-of-patient-engagement-what-is-engagement-and-why-is-it-important [accessed 2019-09-10]
    66. Listowel Wingham Hospitals Alliance. 2017. Quality & Patient Engagement Framework.   URL: http://www.lwha.ca/data/1/rec_docs/1294_LWHA_Quality_Framework_Final_2017.pdf [accessed 2019-07-14]
    67. Agency for Clinical Innovation. 2015. Patient Experience and Consumer Engagement: A Framework for Action.   URL: https://www.aci.health.nsw.gov.au/__data/assets/pdf_file/0005/256703/peace-framework.pdf [accessed 2019-07-14]
    68. Canadian Agency for Drugs and Technologies in Health. 2019. CADTH Framework for Patient Engagement in Health Technology Assessment.   URL: https://www.cadth.ca/cadth-framework-patient-engagement-health-technology-assessment [accessed 2019-07-13]
    69. Health Quality Ontario. UW Departments Web Server. 2016. Choosing Methods for Patient and Caregiver Engagement.   URL: https:/​/depts.​washington.edu/​panport/​resources/​framework-guide/​choosing-methods-patient-and-caregiver-engagement [accessed 2019-07-13]
    70. Canadian Patient Safety Institute. 2018. Engaging Patients in Patient Safety – a Canadian Guide.   URL: https:/​/www.​patientsafetyinstitute.ca/​en/​toolsResources/​Patient-Engagement-in-Patient-Safety-Guide/​Pages/​default.​aspx [accessed 2019-07-13]
    71. NHS England. 2016. Patient Engagement in Patient Safety: A Framework for the NHS.   URL: https:/​/www.​england.nhs.uk/​signuptosafety/​wp-content/​uploads/​sites/​16/​2016/​05/​pe-ps-framwrk-apr-16.​pdf [accessed 2019-07-13]
    72. Government of Prince Edward Island. 2016. Engagement Toolkit.   URL: http://www.gov.pe.ca/photos/original/hpei_engagetool.pdf [accessed 2019-07-13]
    73. Arsand E, Demiris G. User-centered methods for designing patient-centric self-help tools. Inform Health Soc Care 2008 Sep;33(3):158-169. [CrossRef] [Medline]
    74. Manafo E, Petermann L, Mason-Lai P, Vandall-Walker V. Patient engagement in Canada: a scoping review of the 'how' and 'what' of patient engagement in health research. Health Res Policy Syst 2018 Feb 7;16(1):5 [FREE Full text] [CrossRef] [Medline]
    75. Graffigna G, Barello S, Wiederhold B, Bosio AC, Riva G. Positive technology as a driver for health engagement. Stud Health Technol Inform 2013;191:9-17. [CrossRef] [Medline]
    76. Frampton SB, Guastello S, Hoy L, Naylor M, Sheridan S, Johnston-Fleece M. Harnessing evidence and experience to change culture: a guiding framework for patient and family engaged care. NAM Perspectives 2017 Jan 31;7(1):1-38. [CrossRef]
    77. University of Ottawa Heart Institute. 2018. Patient Engagement Framework.   URL: https://www.ottawaheart.ca/patients-visitors/patient-engagement-framework [accessed 2019-07-13]
    78. Abel EA, Shimada SL, Wang K, Ramsey C, Skanderson M, Erdos J, et al. Dual use of a patient portal and clinical video telehealth by veterans with mental health diagnoses: retrospective, cross-sectional analysis. J Med Internet Res 2018 Nov 7;20(11):e11350 [FREE Full text] [CrossRef] [Medline]
    79. Roblin DW, Houston 2nd TK, Allison JJ, Joski PJ, Becker ER. Disparities in use of a personal health record in a managed care organization. J Am Med Inform Assoc 2009;16(5):683-689 [FREE Full text] [CrossRef] [Medline]
    80. Abelson J, Li K, Wilson G, Shields K, Schneider C, Boesveld S. Supporting quality public and patient engagement in health system organizations: development and usability testing of the Public and Patient Engagement Evaluation Tool. Health Expect 2016 Aug;19(4):817-827 [FREE Full text] [CrossRef] [Medline]
    81. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res 2015 Mar;4(2):133-145. [CrossRef] [Medline]
    82. Dukhanin V, Topazian R, DeCamp M. Metrics and evaluation tools for patient engagement in healthcare organization- and system-level decision-making: a systematic review. Int J Health Policy Manag 2018 May 16;7(10):889-903 [FREE Full text] [CrossRef] [Medline]
    83. Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci 2015 Apr 21;10:53 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    EHR: electronic health record
    IT: information technology
    RQ: research question


    Edited by G Eysenbach; submitted 13.05.19; peer-reviewed by T Risling, R Buie, K Fuji, D Alhuwail, R Lee; comments to author 18.06.19; revised version received 14.07.19; accepted 30.07.19; published 08.10.19

    ©Kevin Leung, Drew Lu-McLean, Craig Kuziemsky, Richard G Booth, Sarah Collins Rossetti, Elizabeth Borycki, Gillian Strudwick. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 08.10.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.