This paper is in the following e-collection/theme issue:

Personal Health Records, Patient-Accessible Electronic Health Records, Patient Portals (565) Demographics of Users, Social & Digital Divide (721) Information Seeking, Information Needs (457)

Published on in Vol 23, No 1 (2021): January

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/17782, first published .
The Causal Relationship Between Portal Usage and Self-Efficacious Health Information–Seeking Behaviors: Secondary Analysis of the Health Information National Trends Survey Data

The Causal Relationship Between Portal Usage and Self-Efficacious Health Information–Seeking Behaviors: Secondary Analysis of the Health Information National Trends Survey Data

The Causal Relationship Between Portal Usage and Self-Efficacious Health Information–Seeking Behaviors: Secondary Analysis of the Health Information National Trends Survey Data

Authors of this article:

Jaeyoung Park1 Author Orcid Image ;   Muxuan Liang2 Author Orcid Image ;   Jordan M Alpert3 Author Orcid Image ;   Richard F Brown4 Author Orcid Image ;   Xiang Zhong1 Author Orcid Image
Article Authors Cited by (8) Tweetations (2) Metrics

    Original Paper

    1Department of Industrial and Systems Engineering, University of Florida, Gainesville, FL, United States

    2Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA, United States

    3Department of Advertising, University of Florida, Gainesville, FL, United States

    4Department of Health Behavior and Policy, Virginia Commonwealth University School of Medicine, Richmond, VA, United States

    Corresponding Author:

    Xiang Zhong, DPhil

    Department of Industrial and Systems Engineering

    University of Florida

    PO Box 115002

    Gainesville, FL, 32611

    United States

    Phone: 1 3523922477

    Email: xiang.zhong@ise.ufl.edu


    Background: Patient portals have drawn much attention, as they are considered an important tool for health providers in facilitating patient engagement. However, little is known about whether the intensive use of patient portals contributes to improved management of patients’ health in terms of their confidence in acquiring health information and exercising self-care. There is a lack of randomized trials with these outcomes measured both pre- and postadoption of patient portals.

    Objective: The aim of this study was to examine the causal relationship between the usage of patient portals and patients’ self-efficacy toward obtaining health information and performing self-care.

    Methods: This study was a secondary data analysis that used data from a US national survey, the National Cancer Institute’s Health Information National Trends Survey 5 Cycle 1. Patient portal usage frequency was used to define the treatment. Survey items measuring self-efficacy on a Likert-type scale were selected as the main outcomes, including patients’ confidence in obtaining health information and performing self-care. To establish causality using survey data, we adopted the instrumental variables method. To determine the direction of the causal relationship in the presence of high-dimensional confounders, we further proposed a novel testing framework that employs conditional independence tests in a directed acyclic graph. The average causal effect was measured using the two-stage least squares regression method.

    Results: We showed that frequently using patient portals improves patients’ confidence in obtaining health information. The estimand of the weighted average causal effect was 0.14 (95% CI 0.06-0.23; P<.001). This means that when increasing the portal usage intensity, for instance, from 1-2 times to 3-5 times per year, the expected average increase in confidence level measured on a Likert-type scale would be 0.14. However, we could not conclusively determine the causal effect between patient portal usage and patients’ confidence in exercising self-care.

    Conclusions: The results support the use of patient portals and encourage better support and education to patients. The proposed statistical method can be used to exploit the potential of national survey data for causal inference studies.

    J Med Internet Res 2021;23(1):e17782

    doi:10.2196/17782

    Keywords

    causal inference; instrumental variable; directed acyclic graph; patient portal; self-efficacy 


    Given the growing evidence showing that patient engagement improves health outcomes and reduces health care costs, patient portals have drawn much attention. A patient portal is a secure online health platform linked to a patient’s personal medical record that is 24/7 accessible from any location with an internet connection. Patient portals are considered an important tool for health providers in facilitating patient engagement [1,2]. Characteristics of portal users [3] and barriers to portal adoption [4,5] have been extensively investigated. Regarding the facilitators, it was found that patients who believed the patient portal was empowering demonstrated a higher intention to use [6]. The relationship between the internal use (quantified using engagement measurement) of online health platforms and external growth (its reach), and the social support activities related to users’ participation has been examined as well [7,8]. Research has also focused on whether portal or secure messaging usage can affect the frequency of patients’ office visits [9-15] and appointment adherence [16-19].

    Nonetheless, little is known about whether patient portal usage contributes to the improved management of patients’ health in terms of health literacy, communication, confidence in acquiring health information, and the self-monitoring and self-care of health. Interviews have indicated that patient’s perception of access to online records are associated with a greater focus on their health and more proactive involvement in self-care [20,21]. However, there is a lack of randomized trials with self-efficacy outcomes measured both pre- and postportal adoption. Consequently, a causal link between patients’ portal usage and patients’ health information–seeking behaviors (self-efficacy) has not been formally established.

    Despite the passage of the Health Information Technology for Economic and Clinical Health (HITECH) Act and the recommendations by the US Institute of Medicine [22], the overall adoption rate of portals remains low [23]. A better understanding of the impact of patient portals and the benefit to patients is needed to increase adoption. To this end, we used the Health Information National Trends Survey (HINTS) data [24] to examine the causal relationship between the usage of patient portals and patients’ self-efficacy in obtaining health information and performing self-care.


    Study Setting

    This study was a secondary data analysis that used data from the National Cancer Institute’s HINTS 5 Cycle 1. HINTS 5 Cycle 1 is a cross-sectional survey of a nationally representative sample of US adults used to assess the impact of the health information environment. The survey was conducted from January 2017 through May 2017 using a self-administered mail questionnaire. Out of 10,265 surveys sent out, data were collected from 3285 (32% response rate) respondents [24]. The characteristics of these respondents are summarized in Table S1 in Multimedia Appendix 1.

    Patients’ self-efficacy in obtaining health information and performing self-care were considered as the outcomes. Survey items measuring patient self-efficacy were selected based on the Institute of Medicine’s recommendations for promotion of patient portals to increase quality of care and reduce medical errors [22], as well as claims that health information technology, like portals, could increase patients’ self-efficacy for managing their conditions [25-28]. Perceived confidence was assessed with the following questionnaire items: (1) “Overall, how confident are you that you could get advice or information about health or medical topics if you needed it?” (ConfidentGetHealthInfo, Y1); (2) “Overall, how confident are you about your ability to take good care of your health? (OwnAbilityTakeCareHealth, Y2).”

    These measures were captured on a 5-point Likert-type scale where a higher score indicated greater confidence. Variables on patients’ age, gender, race, ethnicity, marital status, education, employment status, household income, and insurance status were considered to be confounders in the study. Information on patients’ portal activities was elicited by the following questionnaire item: “How many times did you access your online medical record in the last 12 months? (0/1 to 2 times/3 to 5 times/6 to 9 times/10 or more times).”

    Different usage frequencies were considered to be different levels of treatment. For the instrumental variable (IV), we used the following questionnaire item: “Have any of your health care providers including doctors, nurses, or office staff ever encouraged you to use an online medical record? (yes/no).”

    Missing values were sparse and were handled in several ways depending on the variable type: samples missing the outcomes were discarded; for the IVs, missing responses were replaced by “no encouragement”; and for the confounders, missing responses were imputed by the Multivariate Imputation by Chained Equations (MICE) [29] method with the IV and the confounders used as input variables for that method.

    Establishing a Causal Relationship

    We aimed to test the hypothesis that the exposure to patient portals, or intensively using a patient portal, will improve patients’ self-efficacy outcomes. However, since we only used a one-time outcome measurement for each individual, we could not construct the before-after treatment contrast to directly measure the treatment effect. Further, without a randomized experimental design, the causality can be obscured by confounders. To address these issues, we adopted the IV method for causal inference in observational studies [30]. In these studies, IVs are used to adjust for both observed and unobserved confounding effects and to help identify a contrast of outcomes in the absence of temporal data.

    We identified the following item as the binary variable and used it as the IV: “Have any of your health care providers including doctors, nurses, or office staff ever encouraged you to use an online medical record?”

    Encouragement plays an influential role on patients’ use of portals, as care providers’ endorsement is an important factor in the adoption of these tools [6-8,31]. However, provider referrals to use patient portals have long been known to vary by race and socioeconomic status and are thought to be dependent on whether the provider believes that the patient will use the portal [32]. The same is true for the HINTS data we used, where income and education were observed to be associated with recommendation, patient portal usage, and self-efficacy. Thus, the requirement of no confounding between the IV and the outcome might not have been met if encouragement (or other candidate IVs, such as internet savviness) were used as an IV.

    To further determine whether patient portal usage causes an improvement in self-efficacy, or vice versa, we proposed a testing framework that could both address the confounding issue and determine the direction of the causal relationship. Our testing framework was based on causal directed acyclic graphs (DAGs), which are used as a graphical tool to visually represent and understand the concepts of exposure, outcome, causation, and confounding [33]. To identify an appropriate IV, we generalized the criteria of IVs to allow for known (or observable) confounding among the IV, treatment, and outcome; and unknown (or unobservable) confounding between the treatment and outcome [30]. With this separation of confounders, only the known common confounders were essential for examining the causal relationship. To further identify the direction of causality, a testing framework employing conditional independence tests [34,35] was developed. The detailed description of the testing framework can be found in Multimedia Appendix 2.

    Measuring Causal Effect

    The proposed DAG-based testing framework aimed to qualitatively evaluate the causality. To quantify the treatment effect, two-stage least squares (TSLS) regression models were built. As the treatment (portal usage) has multiple levels (eg, 1-2 times and 3-5 times annually are different levels), the traditional average treatment effect is not identifiable. However, Angrist and Imbens [36] showed that an estimand, essentially a weighted average of per-unit average causal effects, is identifiable and can be estimated by TSLS regression models. The two major assumptions therein, independence and monotonicity of ordinal treatment effect, were thus naturally satisfied in our causal framework.


    Portal Enrollment

    After removing the samples with too many missing data, we identified 3198 participants among the 3285 survey respondents. Among these 3198 participants, 1003 (31%) were self-reported patient portal users. For demographic and socioeconomic variables, the user group and the nonuser group had different characteristics (see Table S1 in Multimedia Appendix 1). Patients younger than 65 years old and females were more likely to be patient portal users. Moreover, participants who self-reported as White, married, and non-Hispanic were more likely to be users. In terms of income, higher income was positively correlated with a greater likelihood of being a portal user. Likewise, participants who were employed were more likely to use portals than those who were retired or unemployed. Compared to nonusers, portal users had a higher level of education (eg, undergraduate or postgraduate). Finally, compared to those covered by employer-provided insurance, patients who were covered by private insurance, Medicare, or Medicaid were less likely to enroll in patient portals.

    Encouragement and Portal Usage

    Next, we characterized the users’ portal usage behavior. Of the 1003 portal users, 49% (496) reported using portals 1-2 times in the past 12 months, 31% (313) reported using them 3-5 times, 10% (104) reported using them 6-9 times, and 9% (90) reported using them more than 9 times.

    Of the 3198 participants, 1375 (43%) were encouraged to use patient portals and 1823 (57%) were not. Among the 1375 respondents who were encouraged, 549 (40%), 383 (28%), 267 (19%), and 176 (13%) individuals never used a portal, used a portal 1-2 times, 3-5 times, and more than 5 times, respectively. In contrast, there were 1646 (90%), 113 (6%), 46 (3%), and 18 (1%) participants in the nonencouraged group, respectively. It was evident that the IV and the treatment were significantly associated, which was verified by a chi-square test (P<.001).

    Encouragement and Self-Efficacy

    There were more patients with positive responses for the self-efficacy outcomes in the patient population who were recommended to use portals. The distributions of each outcome variable conditioning on the value of the IV are displayed in Table 1, and chi-square test results are provided. Among the 3198 participants, 3111 and 3165 participants were identified as those who answered questions Y1 and Y2, respectively. For ConfidentGetHealthInfo, individuals in the encouraged group (Ge) were more likely to be completely confident or very confident than those in the nonencouraged group (Gn; Ge: 907/1353, 67%; Gn: 1000/1758, 57%; P<.001). For OwnAbilityTakeCareHealth, Ge patients were slightly more confident in self-care (Ge: 998/1364, 73%; Gn: 1222/1801, 68%; P=.001). All the comparisons suggest that the IV and the outcomes were significantly associated.

    Table 1. Chi-square test results for the association between encouragement to use patient portals and self-efficacy outcomes.
    OutcomesIVa: encouragement to use patient portalsP value

    		Yes, n (%)No, n (%)
    ConfidentGetHealthInfo, Y11353 (100)1758 (100)<.001

    		Not confident at all21 (1.5)50 (2.8)

    		A little confident56 (4.1)108 (6.1)

    		Somewhat confident369 (27.3)600 (34.1)

    		Very confident566 (41.8)607 (34.5)

    		Completely confident341 (25.2)393 (22.4)
    OwnAbilityTakeCareHealth, Y21364 (100)1801 (100).001

    		Not confident at all15 (1.1)31 (1.7)

    		A little confident31 (2.3)79 (4.4)

    		Somewhat confident320 (23.5)469 (26.0)

    		Very confident652 (47.8)815 (45.3)

    		Completely confident346 (25.4)407 (22.6)

    aIV: instrumental variable.

    Causal Relationship Between Patient Portal Usage and Self-Efficacy

    Following the testing procedure described in Multimedia Appendix 2, conditional independence tests were conducted. Test A tested the hypothesis that, given the common confounders and the treatment, the IV is conditionally independent with the outcome. This test examined whether higher self-efficacy is a cause for increased portal usage. Test B tested the hypothesis that the IV and the outcome are conditionally independent, given the common confounders alone. This test examined whether the increase in portal usage is a cause for higher self-efficacy. A P value <.05 was used as the statistical significance level for the overall test. The results are shown in Table 2.

    Table 2. Results of conditional independence tests.
    OutcomesP value
     Test A: Z ⊥ Y ∣ Test B: Z ⊥ Y ∣
    ConfidentGetHealthInfo.28.02
    OwnAbilityTakeCareHealth.42.23

    For ConfidentGetHealthInfo, we could not reject the hypothesis for Test A (overall P=.28) but could reject Test B (overall P=.02), meaning that portal usage did have a causal effect on self-efficacy toward obtaining health information. This relationship is illustrated in Figure 1. However, for OwnAbilityTakeCareHealth, we could not reject the hypothesis for either Test A (overall P=.42) or Test B (overall P=.23). In this case, we could not determine the causal relationship between patients' portal usage and patients’ confidence in exercising self-care.

    ‎
    Figure 1. The causal relationship among encouragement, portal usage, and confidence in seeking health information with the confounders. IV: instrumental variable.
    View this figure

    Treatment Effect of Patient Portal Usage on Self-Efficacy

    A TSLS regression model was built to quantify the treatment effect of portal usage on self-efficacy toward acquiring health information. The estimand of the weighted average causal effect was 0.14 (95% CI 0.06-0.23; P<.001). This indicates that, when increasing the portal usage intensity, for instance, from 1-2 times to 3-5 times per year, the expected average increase in the reported confidence level (measured with a Likert-type scale) would be 0.14. It can be concluded that the more intensely patients engage with portals, the more confident they are in obtaining health information.


    Identifying Causality

    Existing work on survey analysis mainly focuses on the strength of the association between questionnaire items and targeted outcomes [37,38], as there is a lack of understanding regarding the causal path diagram and a lack of methods for determining if the path diagram is congruent with the data. In this study, we developed a new framework that allowed us to detect causality and estimate the magnitude of the treatment effect represented in the path diagram using the HINTS 5 survey. Our framework was able to eliminate the estimation bias due to unmeasured confounders, and guarantee the test efficiency given the limited sample size of the survey. Moreover, the testing framework was robust to the choice of IVs. In addition to using “encouragement” as an IV, we tested internet savviness as an alternative IV and found the conclusion was consistent (see Multimedia Appendix 2).

    Effect of Portal Usage

    Although we should be cautious in interpreting the relationship between using patient portals and self-efficacy as causal, we did observe a co-occurrence of better outcomes and increased patient portal usage intensity. Despite the benefits of patient portals being well documented [39,40], rates of usage remain low [41], and increasing the adoption of patient portals would provide more patients a trusted source of personalized information. Because patients can be easily misinformed by the large amount of inaccurate information online [42], using patient portals allows patients and their caregivers to stay connected with their providers. This occurs through easy access to their own health information and the ability to contact providers via secure messaging if they have questions or concerns. Our framework further confirmed that portal usage does positively affect confidence in obtaining health information. However, we did not observe a similar level of significance to conclude that higher portal usage will lead to higher self-efficacy in performing self-care. It is possible that other confounding factors, such as income and education levels [43], might have had a more powerful effect on the outcome.

    Encouraging Patients’ Portal Use

    Our analysis shows that being encouraged to use patient portals positively affects the intensity of portal usage, which in turn influences people’s confidence in acquiring health information. Other studies have reported that actively using patient portals, such as sending messages to physicians and viewing prescription and lab results, is positively associated with high-quality physician-patient relationships and patients’ confidence to understand health information. It is evident that a trusting physician-patient relationship helps promote healing and remove barriers to obtaining medical information [44]. However, efforts to increase the adoption of patient portals often rely on pamphlets and flyers [45] or hurried conversations by medical staff. Given the potential benefits of patient portals and our findings that encouragement affects usage, we believe that an innovative intervention is necessary to increase patient adoption and usage, which ultimately can lead to significant improvements in patient outcomes.

    Limitations

    The study was conducted using HINTS 5 Cycle 1 data, which were collected in 2017. The study can be improved by combining multiple data sets, including ones published recently [46]. The power of the test can be increased by including more samples, which can potentially help identify the causal relationship between portal usage and self-care. It is worth noting that survey designs are not identical across all three cycles of HINTS 5. For instance, the outcome of ConfidenceGetHealthInfo is available in Cycle 1 (2017) and Cycle 3 (2019) but not in Cycle 2 (2018). Considerable effort will be needed to merge these data sets. Furthermore, the HINTS database contains rich information beyond the variables that were used for this study, and other aspects (eg, health disparities) related to both portal usage and self-efficacy behaviors should be explored.

    The establishment of the causal relationship between portal usage and patients’ self-efficacy demands that encouragement to use portals is not based on patients’ self-efficacy. We have observed that many health care organizations have integrated the facilitation of portal enrollment into their new patients’ registration protocol [47]. In addition, portal use encouragement also occurs during a patient’s interaction with the front desk when addressing billing and appointment scheduling, as patient portals can be an alternative venue to handle these services. These scenarios correspond to the cases that nurses or office staff encourage patients to use an online medical record. These situations account for the majority of the encouragement during patient encounters and thus support the selection of encouragement as an IV. However, as for personal encouragement from a physician, we are unaware of any health care organizations that have policies or incentives for providers to systematically encourage all patients to use portals, meaning that encouragement can be at the providers’ discretion [48]. Therefore, collecting physicians’ input concerning their patterns of encouragement would be valuable for further justifying the validity of using encouragement as an IV. To ensure the rigor of our results, we further tested internet savviness as an alternative IV and found a consistent conclusion (see Multimedia Appendix 2).

    Although the confounders between the treatment and the outcome can be completely unmeasured, we still require the common confounders to be fully observable. The identification of common confounders largely relies on domain knowledge. It is worth noting that the choice does not have to be unique. In our casual diagram, adding variables to the common confounder set still results in a valid choice. However, chi-square tests can be sensitive to the choice of confounders when the sample size is not sufficiently large [49]. Therefore, to make the results more robust, more survey respondents are needed.

    Conclusions

    Since the Affordable Care Act mandated portal usage, enthusiasm for portals has declined; however, this study found that using patient portals improves patients’ confidence in obtaining advice or information about health or medical topics. Our findings thus attest to the benefit of patient portals and to providing better support and education to patients. In addition, our proposed statistical method exploits the potential of using national survey data such as the HINTS program to examine causal effects to obtain new insights. Theoretically, the treatment effect can be heterogeneous based on different patient characteristics. There is thus a need to develop a testing framework that can identify the disparity in causal effects. For justifying the clinical insights identified in this study, we cannot solely rely on patients’ self-reported outcomes, but should also survey physicians on their perception of patients’ self-efficacy. Furthermore, how physicians make encouragement decisions should also be investigated, and a randomized controlled study with pre- and posttreatment outcomes being clearly documented is necessary to fully understand the treatment effect on self-efficacy outcomes.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Table S1.

    PDF File (Adobe PDF File), 56 KB
    ‎
    Multimedia Appendix 2

    Methods.

    DOCX File , 147 KB
    1. Irizarry T, DeVito Dabbs A, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015 Jun 23;17(6):e148 [FREE Full text] [CrossRef] [Medline]
    2. Miller DP, Latulipe C, Melius KA, Quandt SA, Arcury TA. Primary care providers' views of patient portals: interview study of perceived benefits and consequences. J Med Internet Res 2016 Jan 15;18(1):e8 [FREE Full text] [CrossRef] [Medline]
    3. Anthony DL, Campos-Castillo C, Lim PS. Who isn't using patient portals and why? Evidence and implications from a national sample of US adults. Health Aff (Millwood) 2018 Dec;37(12):1948-1954. [CrossRef] [Medline]
    4. Kooij L, Groen WG, van Harten WH. Barriers and facilitators affecting patient portal implementation from an organizational perspective: qualitative study. J Med Internet Res 2018 May 11;20(5):e183 [FREE Full text] [CrossRef] [Medline]
    5. Scott KC, Karem P, Shifflett K, Vegi L, Ravi K, Brooks M. Evaluating barriers to adopting telemedicine worldwide: a systematic review. J Telemed Telecare 2018 Jan;24(1):4-12. [CrossRef] [Medline]
    6. Agarwal R, Anderson C, Zarate J, Ward C. If we offer it, will they accept? Factors affecting patient use intentions of personal health records and secure messaging. J Med Internet Res 2013;15(2):e43 [FREE Full text] [CrossRef] [Medline]
    7. Gopalsamy R, Semenov A, Pasiliao E, McIntosh S, Nikolaev A. Engagement as a driver of growth of online health forums: observational study. J Med Internet Res 2017 Aug 29;19(8):e304 [FREE Full text] [CrossRef] [Medline]
    8. Wang X, Zhao K, Street N. Analyzing and predicting user participations in online health communities: a social support perspective. J Med Internet Res 2017 Apr 24;19(4):e130 [FREE Full text] [CrossRef] [Medline]
    9. Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. JAMA 2012 Nov 21;308(19):2012-2019. [CrossRef] [Medline]
    10. Otte-Trojel T, de BA, Rundall TG, van de Klundert J. How outcomes are achieved through patient portals: a realist review. J Am Med Inform Assoc 2014 Jul;21(4):751-757. [CrossRef] [Medline]
    11. Zhou YY, Leith WM, Li H, Tom JO. Personal health record use for children and health care utilization: propensity score-matched cohort analysis. J Am Med Inform Assoc 2015 Jul;22(4):748-754. [CrossRef] [Medline]
    12. Leveille SG, Mejilla R, Ngo L, Fossa A, Elmore JG, Darer J, et al. Do patients who access clinical information on patient internet portals have more primary care visits? Med Care 2016 Jan;54(1):17-23. [CrossRef] [Medline]
    13. Bavafa H, Hitt LM, Terwiesch C. The impact of e-visits on visit frequencies and patient health: evidence from primary care. Manage Sci 2018 Dec;64(12):5461-5480 [FREE Full text] [CrossRef] [Medline]
    14. Bergmo TS, Kummervold PE, Gammon D, Dahl LB. Electronic patient-provider communication: will it offset office visits and telephone consultations in primary care? Int J Med Inform 2005 Sep;74(9):705-710. [CrossRef] [Medline]
    15. Meng D, Palen TE, Tsai J, McLeod M, Garrido T, Qian H. Association between secure patient-clinician email and clinical services utilisation in a US integrated health system: a retrospective cohort study. BMJ Open 2015 Nov 09;5(11):e009557 [FREE Full text] [CrossRef] [Medline]
    16. Horvath M, Levy J, L'Engle P, Carlson B, Ahmad A, Ferranti J. Impact of health portal enrollment with email reminders on adherence to clinic appointments: a pilot study. J Med Internet Res 2011;13(2):e41 [FREE Full text] [CrossRef] [Medline]
    17. Lin C, Mistry N, Boneh J, Li H, Lazebnik R. Text message reminders increase appointment adherence in a pediatric clinic: a randomized controlled trial. Int J Pediatr 2016;2016:8487378 [FREE Full text] [CrossRef] [Medline]
    18. Mendel A, Chow S. Impact of health portal enrolment with email reminders at an academic rheumatology clinic. BMJ Qual Improv Rep 2017;6(1):1-9 [FREE Full text] [CrossRef] [Medline]
    19. Zhong X, Park J, Liang M, Shi F, Budd PR, Sprague JL, et al. Characteristics of patients using different patient portal functions and the impact on primary care service utilization and appointment adherence: retrospective observational study. J Med Internet Res 2020 Feb 25;22(2):e14410 [FREE Full text] [CrossRef] [Medline]
    20. Zarcadoolas C, Vaughon WL, Czaja SJ, Levy J, Rockoff ML. Consumers' perceptions of patient-accessible electronic medical records. J Med Internet Res 2013;15(8):e168 [FREE Full text] [CrossRef] [Medline]
    21. Dhanireddy S, Walker J, Reisch L, Oster N, Delbanco T, Elmore JG. The urban underserved: attitudes towards gaining full access to electronic medical records. Health Expect 2014 Oct;17(5):724-732. [CrossRef] [Medline]
    22. Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century. Washington (DC): National Academies Press (US); 2001:A. [CrossRef] [Medline]
    23. Fraccaro P, Vigo M, Balatsoukas P, Buchan IE, Peek N, van der Veer SN. Patient portal adoption rates: a systematic literature review and meta-analysis. Stud Health Technol Inform 2017;245:79-83. [Medline]
    24. About HINTS. The Health Information National Trends Survey.   URL: https://hints.cancer.gov/about-hints/learn-more-about-hints.aspx [accessed 2019-08-29]
    25. Dorr D, Bonner LM, Cohen AN, Shoai RS, Perrin R, Chaney E, et al. Informatics systems to promote improved care for chronic illness: a literature review. J Am Med Inform Assoc 2007;14(2):156-163 [FREE Full text] [CrossRef] [Medline]
    26. Rigby M, Georgiou A, Hyppönen H, Ammenwerth E, de Keizer N, Magrabi F, et al. Patient portals as a means of information and communication technology support to patient-centric care coordination - the missing evidence and the challenges of evaluation. A joint contribution of IMIA WG EVAL and EFMI WG EVAL. Yearb Med Inform 2015 Aug 13;10(1):148-159 [FREE Full text] [CrossRef] [Medline]
    27. Allen A, Des Jardins TR, Heider A, Kanger CR, Lobach DF, McWilliams L, et al. Making it local: beacon communities use health information technology to optimize care management. Popul Health Manag 2014 Jun;17(3):149-158 [FREE Full text] [CrossRef] [Medline]
    28. McCarrier KP, Ralston JD, Hirsch IB, Lewis G, Martin DP, Zimmerman FJ, et al. Web-based collaborative care for type 1 diabetes: a pilot randomized trial. Diabetes Technol Ther 2009 Apr;11(4):211-217 [FREE Full text] [CrossRef] [Medline]
    29. Buuren SV, Groothuis-Oudshoorn K. mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software 2010;45(3):1-67 [FREE Full text] [CrossRef]
    30. Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causal inference. Stat Med 2014 Jun 15;33(13):2297-2340 [FREE Full text] [CrossRef] [Medline]
    31. Shah SD, Liebovitz D. It takes two to tango: engaging patients and providers with portals. PM&R 2017 May;9(5S):S85-S97. [CrossRef] [Medline]
    32. Ancker JS, Barrón Y, Rockoff ML, Hauser D, Pichardo M, Szerencsy A, et al. Use of an electronic patient portal among disadvantaged populations. J Gen Intern Med 2011 Oct;26(10):1117-1123 [FREE Full text] [CrossRef] [Medline]
    33. Pearl J. Causal diagrams for empirical research. Biometrika 1995;82(4):669-688. [CrossRef]
    34. Conditional Independence. PennState.   URL: https://online.stat.psu.edu/stat504/lesson/5/5.4/5.4.4
    35. Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011 May;46(3):399-424 [FREE Full text] [CrossRef] [Medline]
    36. Angrist JD, Imbens GW. Two-stage least squares estimation of average causal effects in models with variable treatment intensity. Journal of the American Statistical Association 1995 Jun;90(430):431-442. [CrossRef]
    37. Hou J, Shim M. The role of provider-patient communication and trust in online sources in internet use for health-related activities. J Health Commun 2010;15 Suppl 3:186-199. [CrossRef] [Medline]
    38. Hong H, Oh HJ. The effects of patient-centered communication: exploring the mediating role of trust in healthcare providers. Health Commun 2020 Apr;35(4):502-511. [CrossRef] [Medline]
    39. Reed ME, Huang J, Brand RJ, Neugebauer R, Graetz I, Hsu J, et al. Patients with complex chronic conditions: health care use and clinical events associated with access to a patient portal. PLoS One 2019;14(6):e0217636 [FREE Full text] [CrossRef] [Medline]
    40. Stewart MT, Hogan TP, Nicklas J, Robinson SA, Purington CM, Miller CJ, et al. The promise of patient portals for individuals living with chronic illness: qualitative study identifying pathways of patient engagement. J Med Internet Res 2020 Jul 17;22(7):e17744 [FREE Full text] [CrossRef] [Medline]
    41. Tsai R, Bell EJ, Woo H, Baldwin K, Pfeffer MA. How patients use a patient portal: an institutional case study of demographics and usage patterns. Appl Clin Inform 2019 Jan;10(1):96-102 [FREE Full text] [CrossRef] [Medline]
    42. Swire-Thompson B, Lazer D. Public health and online misinformation: challenges and recommendations. Annu Rev Public Health 2020 Apr 02;41:433-451. [CrossRef] [Medline]
    43. Greenland S. Quantifying biases in causal models: classical confounding vs collider-stratification bias. Epidemiology 2003 May;14(3):300-306. [Medline]
    44. Thom DH, Hall MA, Pawlson LG. Measuring patients' trust in physicians when assessing quality of care. Health Aff (Millwood) 2004;23(4):124-132 [FREE Full text] [Medline]
    45. Bronsburg S, Durity M, Ramim M. An exploratory study of patient portal adoption and use in multiple clinical specialty departments. OJAKM 2019 Oct 10;7(1):86-101. [CrossRef]
    46. HINTS 5 overview and schedule. The Health Information National Trends Survey.   URL: https://hints.cancer.gov/about-hints/data-collection-schedule.aspx [accessed 2020-08-29]
    47. Ancker JS, Nosal S, Hauser D, Way C, Calman N. Access policy and the digital divide in patient access to medical records. Health Policy and Technology 2017 Mar;6(1):3-11. [CrossRef]
    48. Grossman LV, Masterson Creber RM, Benda NC, Wright D, Vawdrey DK, Ancker JS. Interventions to increase patient portal use in vulnerable populations: a systematic review. J Am Med Inform Assoc 2019 Aug 01;26(8-9):855-870. [CrossRef] [Medline]
    49. McHugh ML. The chi-square test of independence. Biochem Med (Zagreb) 2013;23(2):143-149. [Medline]

    ‎
    DAG: directed acyclic graph
    HINTS: Health Information National Trends Survey
    HITECH: Health Information Technology for Economic and Clinical Health
    IV: instrumental variable
    MICE: Multivariate Imputation by Chained Equations
    TSLS: two-stage least squares

    Edited by G Eysenbach, Q Zeng; submitted 14.01.20; peer-reviewed by A Nikolaev, N Benda, W van Harten, D Pförringer, V Osadchiy, C Luckhurst; comments to author 28.08.20; revised version received 25.09.20; accepted 21.12.20; published 27.01.21

    Copyright

    ©Jaeyoung Park, Muxuan Liang, Jordan M Alpert, Richard F Brown, Xiang Zhong. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 27.01.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.