Internationally, there has been an increasing effort to engage patients and consumers in their own health care using information and communication technology. The COVID-19 pandemic significantly stimulated the adoption and use of information and communication technology in primary care and outpatient clinics to facilitate remote visits, distant monitoring, and communication during the period of social distancing, particularly for patients living with chronic conditions [1-4]. These current events will further motivate various stakeholders to revisit the importance of eHealth tools, including electronic patient portals. A recent example from Canada is the province of Alberta that launched 2 patient portals in 2019, with a plan to gradually expand functionalities and patient engagement [5,6]. Since April 2020, patients and providers tested for COVID-19 across Alberta are able to access their test results online via MyHealth Records [5].

Countries such as Canada, the United Kingdom, and the United States have created national consumer digital health strategies and programs to encourage greater patient and consumer interactions with their health care providers through a variety of digital health solutions such as the patient portal [7-9]. For example, Infoway, an organization promoting Canada’s health strategy, has funded several portal implementation projects over the last decade and produced benefit evaluation reports [8].

Patient portal is a digital health tool managed by a health organization to provide patients with secure online access to their own health information such as laboratory results, doctor’s notes, and medication lists; care services such as appointment booking and reminders; and communication with their health care providers via secure messaging from anywhere via the internet [10]. It is also known as a tethered personal health record, which is a web-based interface linked to an electronic health record (EHR) where patients can view and interact with their health care data and providers [11].

Despite widely acknowledged portal benefits, the empirical evidence on patient portals reflects important challenges and context-dependent outcomes of portal use. For example, a recent review examining the behavioral and clinical outcomes associated with portal use reported improved patient understanding of their health conditions and medication adherence while noticing modest or no effects on diabetes and hypertension biophysiological indicators [12]. Our previous review demonstrated that portal technology may inadvertently create health equity concerns by not paying sufficient attention to the social determinants of health during portal implementation [13].

Research on patient portals, including primary studies (PSs), systematic reviews (SRs), and meta-level reviews, has been rapidly growing. Most SRs focus on specific health conditions [14], patient populations [15], aspects of portal use and its effects or impact [16-18], or on a select study design, for example, randomized controlled trials (RCTs) using a portal as an intervention [16]. To date, 2 meta-level reviews have been published that summarize the findings on patient portals reported in SR papers [19,20]. van Mens et al [19] used the Clinical Adoption Framework to “map relationships between determinant and outcome category” from 19 SRs retrieved up to early 2018. Among the limitations of their review, van Mens et al [19] named inadvertent inclusion of duplicate PSs and the impossibility of evaluating the strength of evidence. The other review of 20 SRs summarized the methods (ie, study design), types of effects, and benefits of digital health interventions up to 2016 [20].

In addition to focusing on portal technology, both meta-level reviews [19,20] and several SRs [21-23] reported combined findings on a variety of eHealth tools, which may limit the reader’s ability to discern portal-specific effects. Further, SRs may not always explicitly assess the quality of included PSs, thus potentially giving equal weight to findings characterized by various degrees of empirical support. When quality appraisals are included, the focus is often on RCTs and not broadened to other study designs [12,24]. Further, to the best of our knowledge, no published reviews summarizing evidence on patient portals have evaluated the strength of generated evidence as the concluding step of their review process. The limited quality assessments of the included studies and the absence of evaluations of the strength of outcomes or evidence offer readers little guidance on interpreting blanket statements of mixed and inconclusive results. To sum up, the various types of portal reviews that have been conducted to date present some substantive and methodological limitations mentioned earlier, thus providing an opportunity for our umbrella review.

Our contributions are threefold. The first is substantive: this umbrella review comprehensively consolidates the current state of evidence about patient portals. The second is methodological: we included a wide range of high-quality SRs that were specifically focused on portal technology, eliminated duplicate PSs, and appraised the quality of umbrella review quantitative and qualitative evidence using modified GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) and CERQual (Confidence in the Evidence from Review of Qualitative Research) criteria, thus presenting the relative strength of each umbrella review finding. The third contribution is knowledge translation, where our findings in the form of an evidence map can provide guidance for organizations in their patient portal adoption efforts. This is especially important in the Canadian context as many jurisdictions are actively pursuing patient portals at this time.

The objectives of this umbrella review are to summarize the current state of evidence in patient portals based on published SRs and to create an evidence-based knowledge translation tool for the adoption of this technology. The questions addressed in this umbrella review are as follows:

Our methodology is detailed in a published protocol [25] registered and updated in PROSPERO (International Prospective Register of Systematic Reviews; PROSPERO registration number CRD42018096657). We employed the Joanna Briggs Institute (JBI) umbrella review method [26] with modifications [25]. Overall, our umbrella review attempted to adhere to best practice methodological recommendations outlined by Pollock et al [27] and Smith et al [28].

The population, objectives, design, and context of the 14 reviews are described below and further summarized in Multimedia Appendix 2. All 14 reviews were published between November 2012 and November 2019, 11 of which were published in 2015 or later. With duplicates removed, 280 unique PSs were identified across the 13 aggregation reviews, resulting in 1075 unique findings. We handled the findings of the only configuration review [18] holistically, without disintegrating into separate components. This realist review [18] proposed mechanisms for achieving portal outcomes.

Portal features were reported inconsistently and unsystematically across reviews (a summary of the portal features described in the reviews is provided in Multimedia Appendix 2). Two SRs itemized and compared specific patient portal features reported across each PS [31,33]. Other SRs summarized portal features in their results section or appendices as an overview of the intervention, a collective summary sentence, or in reference to individual findings. Some reviews did not report specific portal features in their findings, but described portal functions in general, in the introductory section.

In total, we identified 41 portal features, with a range across reviews from 3 [30] to 25 [31]. Secure messaging or communication and appointment booking were the 2 features mentioned in all reviews. Other common features that we identified were access to laboratory and test results, visit summaries, and medication renewals. In contrast, portal features that allowed patients to generate data through care plans, patient self-assessment tools, journals, and the ability to edit data were mentioned less frequently. Frequently used features in a pediatric portal included immunization records, secure messaging, and appointment scheduling [15]. The SR by Kelly et al [33] on inpatient portals highlighted patients’ desire to be able to view their daily schedule and information on medication dose, frequency, timing, administration, route, and side effects.

Seven reviews mentioned features that patients viewed as desirable but not commonly offered. These included proxy access, medication glossaries with photos, medication side effects and instructions, care goals or plans with feedback, symptom tracking, videoconferencing, portal access through onsite kiosks, voice recognition for older adults, and text messaging for quality assurance service [15,18,24,29-31,33]. Specific features desired by patients in an inpatient setting included hospital room number; health care provider names or photos; medical information on condition and what will happen next; recovery goals; and access to physician notes, operative reports, and test results [33].

Tables 1 and 2 display the umbrella review finding statements and the evidence profile. To understand how the strength of the evidence and the confidence in the evidence were evaluated, refer to Multimedia Appendix 3. We found few examples of high confidence in the evidence and no examples of high strength of evidence. High confidence means that additional studies are unlikely to generate new findings on account of the topic being relatively well researched, SRs being of high quality, and the findings representing the phenomenon accurately. Moderate strength of evidence indicates that the finding is likely but there are some deficiencies in the current evidence [41]. Moderate confidence in the evidence indicates that the findings reasonably represent the phenomenon [43]. Low to very low confidence indicates that the topic is under-researched, studies have methodological weaknesses or inconsistent findings, and new studies are likely to generate useful findings that can contradict existing evidence. Our umbrella review findings should be interpreted in this context, with the amount and quality of existing research represented in the included SRs and the consistency of findings playing an important role in the rating of evidence.

Our evidence map for patient portal adoption is presented in Figure 3. The map is a visual knowledge translation tool of the current state of evidence of patient portal use and impact. In this figure, the image of the 4 CAMM stages is from Price et al [45] and the 5 columns provided have been organized by the domain, and outcome or findings listed in Tables 1 and 2. In the remaining sections of our results, we present the umbrella review findings through the 4 stages of the map, while drawing attention to select strength of evidence and confidence in the evidence ratings.

Data included in our mixed methods and quantitative descriptive extraction tables and not subject to the evidence rating provided additional support for the quantitative and qualitative umbrella review findings discussed earlier (Multimedia Appendix 4). In addition, noteworthy findings from this group of studies that are not found in our evidence tables include providers preferring emails focused on simple, self-limiting problems [24]; the importance of tailored messages sent to patients [29] and proxy access (family member access and caregiver access to information in the portal) [15,30,31,33]; and lower uptake of patient portals than initially anticipated [15,24]. Of note, proxy access and portal uptake are addressed only in a few PSs.

To the best of our knowledge, our umbrella review demonstrates the first attempt to adapt GRADE and CERQual processes and to develop GRADE-UR and CERQual-UR tools for evaluating evidence generated in an umbrella review. Moreover, our combined approach of evaluating the evidence and application of CAMM at the umbrella review level provides a novel approach for analyzing outcomes. In particular, we demonstrate how this approach can provide a more nuanced understanding of the evidence for common findings generated in portal research.

Our umbrella review provides an evidence map based on CAMM and the consolidated summary on the current state of evidence on patient portals and can be used to inform all stages of portal adoption. The map should be used in conjunction with the evidence tables to understand the strength of the available empirical support for different factors influencing patient portal adoption, use, and outcomes. In the next four paragraphs, we present key umbrella review findings and elaborate on how the map can be applied to address current knowledge gaps and across research, industry, policy, and practice.

The temporal aspect of CAMM suggests that portal adoption per se is not a guarantee of its effectiveness and that it should not be evaluated at a single point in time; rather, there are transitions from interest to registration or enrollment, activation, and then to use or utilization ending with empowerment in the best-case scenario. In our umbrella review, we found enrollment and use often being conflated, thus blurring the line between these 2 separate, yet very significant, dimensions of portal adoption.

We color coded different factors included in the evidence map to visually represent the strength of the current evidence. The colors can reveal not only existing knowledge gaps but also methodological assumptions made in some research on portals. For example, the evidence for the statement portals change provider workload was rated as very low, indicating that further research on this topic is likely to produce useful findings. However, the tables summarizing our findings also show that this topic has been extensively researched, but the studies exhibited great variability in the direction of the findings. This incongruence in the findings about changes to provider workload reveals a complex interplay of factors mobilized when technology is introduced into clinical practice and patients’ homes. Neither the types of studies that control for these factors (eg, RCTs) nor the study designs that simply explore participants’ satisfaction and perception are capable of shedding light on local organizational contexts that are often responsible for divergent portal outcomes.

The ideal use of the evidence map would be that each stakeholder looks across adoption stages and recognizes the interdependency among different factors, as the following examples suggest (italics indicate factors within the map). For example, for industry to support the promised long-term vision of improving clinical and patient-oriented outcomes, an easy-to-use portal design is required that does not introduce usability barriers at a later stage. In the policy realm, there is a need for governance guidelines that address both patients’ and providers’ privacy and security concerns to encourage trusting patient-provider relationships.

In addition, the evidence map reveals the areas of differing values that need to be considered within practice to achieve successful portal adoption. For example, patients are interested in the communication afforded by patient portals, whereas providers are concerned about the increased workload introduced by the new channels for communication. Further, portals are promoted as a tool for patient empowerment, whereas providers are concerned about the loss of control over information. With portals designed to serve more than one user group and to support their sometimes-divergent agendas, the success of portal technology hinges on acknowledging these different values and addressing these differences through the engagement of all relevant user groups.

There are a number of strengths to this umbrella review. First, the elimination of duplicate PSs from the included SRs provides a more accurate account of the reported findings. Second, the application of vote-counting, GRADE-UR, and CERQual-UR allows the direction, strength, and confidence of the evidence to be quantified and compared. Third, the evidence-based knowledge translation tool offers practical guidance to those involved in the planning, implementation, and support of patient portals. The evidence map seems promising as it helps to cast the use and impact of patient portals over time across preadoption, early adoption, and mature adoption stages while summarizing both key known success factors strongly supported by research and areas with low evidence base where more research is needed.

There are possible limitations to our search strategy. First, the literature on patient portals is varied in its coverage of eHealth modalities, inconsistent in defining portals, and evolving. Our search and selection strategy could have missed reviews that should be included. Second, we only focused on reviews published in English; those in other languages could have been missed. We also relied on the SR authors' identification of patient portals. However, in reviewing the SRs' reference list, we discovered that some SRs had included PSs that were broader in scope than the SR definition of patient portals. We accounted for these occasional discrepancies between the intended and actual SR focus during our GRADE-UR and CERQual-UR evaluation.

Our application of GRADE and CERQual at the meta-synthesis level could have been flawed, as there is little or no guidance available on how to appraise evidence synthesized across several SRs. In particular, we used SR authors’ reporting of statistical significance and P values to evaluate the strength of evidence. Many of the PSs included in the GRADE-UR evaluation had a limited sample size, and reporting on the power calculation of these studies was lacking; therefore, we were unable to evaluate these claims of statistical significance.

Similarly, we relied on SR authors’ reporting of the PS findings. When we suspected possible errors in SR authors’ reporting of findings, we consulted the original PS; in these cases, consensus among all umbrella review researchers was sought to determine how to address these individual discrepancies. However, this process was not exhaustive, and we did not cross-check all the findings between SRs and PSs.

At the time of analyzing the included SRs, we noticed a certain tendency for not recognizing the findings from qualitative research as evidence worthy of being evaluated for its strength, which may reinforce the hierarchy of what is considered evidence. This observation might be explained by the lack of appropriate tools such as CERQual, which was developed recently and presents a counterpart to GRADE. The notion of evidence in the context of portal research should not be limited to a narrowly conceived evidence stemming from controlled studies. The role of RCTs is well recognized, and the strength of this design is acknowledged [20]; however, the findings and issues raised in our umbrella review and other reviews [46] call for a broader conceptualization of evidence. There are promising emergent qualitative designs that focus on the social, organizational, political, policy, and local context through emergent, ethnographic, and co-design approaches [47,48].

When examining portal-related outcomes, comparative studies to date have focused on biomedical measures for people with diabetes or hypertension and patient empowerment scores for general patient populations. We recommend further research with people living with other chronic conditions and comparative measures that communicate patients' values and perspectives [49]. This will involve greater integration of patient-oriented measures that can evaluate the outcomes that are of greatest interest to patients and experiences with their portal-enabled health care encounters.

Unsystematic reporting of portal features in the included SRs may reveal a mistaken assumption in some patient portal research that portal features are inconsequential for the outcomes. In comparative research, this can lead to a faulty comparison of portals that may be quite different from each other. We recommend that researchers report actual portal functionalities and technology characteristics. Ammenwerth et al [50] offer a simple yet useful portal taxonomy that includes 7 functionalities (access, remind, request, communicate, share, manage, and educate), which can be easily used when describing portal features to support comparisons across settings and studies.

Consistent with the findings of other SRs not included in our umbrella review, we found that the most extensive areas of research and evidence were on patient-related factors, namely, common barriers and facilitators [22,51], clinical and behavioral outcomes [20], and the role of patient demographic factors in portal adoption [19,20,22,46,51]. Health care provider–related factors were primarily focused on provider concerns [22,46,51], with a lack of examples on how these concerns can be addressed. SR reporting of a lack of guidelines and business models to guide portal implementation was the only statement we identified under system-related factors. Our umbrella review extends the understanding of these patient-, provider-, and system-related factors by reporting and rating the evidence at a meta-level. In addition, we identified current evidence gaps related to proxy access, portal uptake, and most significant to our review, the need for theoretical frameworks sensitive to system-level factors. We aimed to address this theoretical gap in applying CAMM to the development of our evidence map for patient portal adoption.

Our umbrella review offers an organized knowledge translation tool on what is known about patient portals, the quality of the available evidence, and areas that require further work. The evidence map can be used to inform planning, implementing, and supporting the adoption of patient portals across research, industry, policy, and practice. Through our GRADE-UR and CERQual-UR approach, we demonstrated not only how to consolidate findings from SRs, including PSs of various designs, but also how to evaluate the strength and confidence in the evidence of findings from quantitative and qualitative studies. For many of the umbrella review findings, the quality of the evidence was rated as low. This suggests at least two interrelated conclusions. For many identified factors playing a role in portal success, their interactions and underlying mechanisms, especially over time, are still mostly unknown and invite new research. Along with this, we need a broader conceptualization as to what constitutes evidence. This calls for study designs and theoretical perspectives attentive to the contextual complexity of portal adoption.