JMIR Publications

Journal of Medical Internet Research

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 17.10.17 in Vol 19, No 10 (2017): October

This paper is in the following e-collection/theme issue:

    Original Paper

    The Association of Patient Factors, Digital Access, and Online Behavior on Sustained Patient Portal Use: A Prospective Cohort of Enrolled Users

    1VA Maine Healthcare System, Augusta, ME, United States

    2VA Portland Health Care System, Center to Improve Veteran Involvement in Care, Portland, OR, United States

    3Connected Care Office, Veterans Health Administration, U.S. Department of Veterans Affairs, Washington, DC, United States

    4Health Policy Research Group, University of Oregon, Eugene, OR, United States

    5Bedford VA Medical Center, Center for Healthcare Organization and Implementation Research, Bedford, MA, United States

    *these authors contributed equally

    Corresponding Author:

    Susan S Woods, MD, MPH

    VA Maine Healthcare System

    1 VA Center

    Augusta, ME, 04330

    United States

    Phone: 1 207 623 8411

    Email:


    ABSTRACT

    Background: As electronic health records and computerized workflows expand, there are unprecedented opportunities to digitally connect with patients using secure portals. To realize the value of patient portals, initial reach across populations will need to be demonstrated, as well as sustained usage over time.

    Objective: The study aim was to identify patient factors associated with short-term and long-term portal usage after patients registered to access all portal functions.

    Methods: We prospectively followed a cohort of patients at a large Department of Veterans Affairs (VA) health care facility who recently completed identity proofing to use the VA patient portal. Information collected at baseline encompassed patient factors potentially associated with portal usage, including: demographics, Internet access and use, health literacy, patient activation, and self-reported health conditions. The primary outcome was the frequency of portal log-ins during 6-month and 18-month time intervals after study enrollment.

    Results: A total of 270 study participants were followed prospectively. Almost all participants (260/268, 97.0%) reported going online, typically at home (248/268, 92.5%). At 6 months, 84.1% (227/270) of participants had visited the portal, with some variation in usage across demographic and health-related subgroups. There were no significant differences in portal log-ins by age, gender, education, marital status, race/ethnicity, distance to a VA facility, or patient activation measure. Significantly higher portal usage was seen among participants using high-speed broadband at home, greater self-reported ability using the Internet, and routinely going online. By 18 months, 91% participants had logged in to the portal, and no significant associations were found between usage and demographics, health status, or patient activation. When examining portal activity between 6 and 18 months, patients who were infrequent or high portal users remained in those categories, respectively.

    Conclusions: Short-term and long-term portal usage was associated with having broadband at home, high self-rated ability when using the Internet, and overall online behavior. Digital inclusion, or ready access to the Internet and digital skills, appears to be a social determinant in patient exposure to portal services.

    J Med Internet Res 2017;19(10):e345

    doi:10.2196/jmir.7895

    KEYWORDS



    Introduction

    Hopes for personal health records (PHRs) and patient portals run high. The spread of these technologies has been propelled by the US Electronic Health Record Incentive Program and Meaningful Use [1], and consumers’ desire to access their health information, email their providers, and request appointments online [2]. As a result, patients and health systems alike increasingly view digital health services as important means to enhance patient access, drive self-care, and improve the care experience [3,4].

    As access to broadband Internet grows and citizens increasingly connect to the Internet using mobile devices, there are unprecedented opportunities to expand remote patient services. While the digital divide in the United States narrows, some gaps are likely to persist. Internet use among US adults rose from 64% in 2005 to 84% in 2015, with 74% of adults with low incomes and 78% of rural residents currently online [5]. However, lower use continues among adults with less than a high school education (66%), and those aged 65 years and older (58%) [5]. Disparities have also been seen when comparing patients who do and do not register for a patient portal. Older patients, those with lower levels of education or income, and African-Americans and Hispanics appear less likely to register for portals [6-9]. As health information technology gains sophistication and health systems offer more virtual services, these demographic differences can impact health equity and outcomes.

    Evidence on the impact of patient portal use is mixed, depending upon which functions are available and the measures that are examined. Most studies show that portals offer convenience and enhanced patient satisfaction, with users feeling more in control of their care [10,11]. Patients who use secure emails, refill medications, and access their clinical notes and test results may improve self-care and increase adherence to treatments [12-16]. In one study of portal use by patients with diabetes and hypertension, using a secure email and refilling medications online were associated with improved outcomes [17]. Overall, portals show great promise as a key adjunct to, or at times a substitute for, traditional care and communication.

    Studying the use of patient portals has been challenging. Measures of portal use vary across studies, and differing patient populations tend to be examined. Patients need to be aware of portal availability, register or enroll, initially sign in online, and ultimately have the capacity to use the portal as their needs arise. Portal registrants, a group frequently presented in studies, may not accurately reflect those using a portal. Additionally, the value of portals to patients and health systems is more complex than measuring usage. However, the repeated and sustained use of portals may serve as a proxy for benefit. Understanding the value of portals, therefore, must at least demonstrate initial reach across a patient population and show sustained usage over time.

    Although a digital divide with broadband Internet does exist, adults who are online and have a chronic condition are more likely than other online adults to search for health information, read online reviews about medications and treatments, or use online peer support [18,19]. What is less clear is the degree to which disparities exist among patients registered for a portal who log in more often compared to those with less frequent portal use. In one study, patients refilling medications online were found to have fewer prescription interruptions, and this finding persisted across all racial and ethnic subgroups [20]. Conversely, racial disparities in portal use were found among patients with human immunodeficiency virus, with non-white participants having lower portal use to monitor their treatment outcomes [21].

    Studies examining portal usage have largely been cross-sectional or evaluated matched cohorts. We conducted an exploratory study, prospectively following patients newly enrolled to use a portal, and explored factors associated with portal usage over time. The aim of the study was to identify patient factors associated with portal usage at 6 months and 18 months after initial enrollment (among those completing verification in person). We sought to describe short-term and long-term portal usage, and to examine Internet-related or digital divide issues among patients who visited the portal less frequently compared to those who did so more often.


    Methods

    Design and Setting

    Our study was conducted at a large Department of Veterans Affairs (VA) health care facility, where there is a requirement for a patient to complete in-person identity proofing in order to access all portal functions. While this step could prove cumbersome for some, it offered a natural opportunity to examine patients who presumably had some Internet access and were motivated to register for full portal access. We prospectively followed a cohort of VA patients who completed the in-person identity proofing process to fully access the VA’s secure portal, My Health e Vet (MHV). At the time of the study, patients who verified their identity could use MHV to refill medications, securely email their providers, receive tailored wellness reminders, view appointments, access laboratory test results through the Blue Button feature, and search the VA health education library [22].

    Patients eligible for the study received care at the VA Portland Health Care System (VAPHCS) and completed identity proofing for MHV. Each VA facility supports staff that assist Veterans in completing the process. At the study site, the MHV office was adjacent to the Outpatient Pharmacy located in the main lobby of the medical center. MHV staff informed patients about the study by handing them an informational flyer. When a patient was interested in the study, a “warm hand-off” was completed: MHV staff contacted research staff by phone or instant messaging, who immediately met with the patient. The study coordinator (ES) consented and enrolled all subjects and conducted a health literacy assessment. All baseline survey questions were completed on paper at the time of enrollment or within 30 days, and returned by mail in a prestamped envelope. Follow-up surveys were conducted at 6 months to collect patient-reported portal usage. Participants were emailed a link to the survey, with two reminders sent. Due to the MHV staff’s busy workday, we were unable to collect data on all patients who were informed about the study. Participants received US $20 for the baseline and US $30 for follow-up surveys. The study received approval from the VAPHCS Institutional Review Board.

    Measures

    The primary outcome measure was the frequency of portal log-ins during 6-month and 18-month time intervals after study enrollment. Log-in data were obtained on all study subjects from the VA’s national Connected Health Office, using MHV Activity Reports. These data were an objective measure of portal use, and listed the total number of successful and unsuccessful (incomplete) MHV log-ins for each study subject.

    The study collected information on factors potentially associated with portal usage. Baseline patient measures included: demographics; distance to a VA facility; and technology access and use, including type and location of Internet connection, comfort with computers, and regular online activity (eg, emailing, shopping, social networking, and searching for health information). To assess patients’ overall use of the Internet, a composite Internet Use Behavior measure was created that included 11 online activities: accessing the Internet; email; general search; health search; shopping; banking; geolocation; visiting any social network site; registration on any site; posting on any site; and using Facebook, MySpace, or LinkedIn. High Internet use represented going online for at least 7 activities. Health literacy was approximated using the Short Form Functional Health Literacy Assessment (S-TOFHLA) [23]. Level of patient engagement in their health and health care was measured using the Patient Activation Measure (PAM), which is a validated, 13-item instrument [24]. Subjects were asked to self-report their health status and presence of diabetes, hypertension, heart disease, arthritis, depression, tobacco use, asthma, or chronic pulmonary disease. Questions also solicited beliefs about PHRs, how subjects learned about MHV, and prior training on the use of MHV. Subjects were asked about their expectations for using MHV and its available functions.

    Analysis

    Baseline characteristics and outcomes were described using frequencies and percentages for the categorical outcomes. Due to the nonnormal distribution of log-ins over time, portal usage was categorized into 4 distinct categories at each time interval. For 6 months, categories included: 0 or 1 log-in, 2 to 5 log-ins, 6 to 11 log-ins, and 12 or more log-ins. Similarly, at 18 months the categories included: 0 to 2 log-ins, 3 to 17 log-ins, 18 to 35 log-ins, and 36 or more log-ins. These 4 categories of log-ins approximately corresponded to portal use frequencies of never/rare use, less than monthly, once or twice per month, and more than twice per month, respectively. To examine the association between frequency of patient portal use and individual patient characteristics, perceptions, and self-reported behavior, we used univariate Chi-square tests. The associations examined frequency of log-ins during the 6-month period after enrollment, the 18-month period after enrollment, between 6 and 18 months, and all patient factors, including demographics (eg, education, income), self-reported health status, PAM score, and S-TOFHLA score. Statistical significance was set at the alpha=0.05 level. All analyses were completed using Stata 14.0 [25].


    Results

    A total of 270 participants were enrolled from December 13, 2010 to January 24, 2012 and completed baseline surveys. Portal usage was followed for 18 months after the date of consent, from mid-June 2013 through the end of July 2013. A total of 230 participants (230/270, 85.2%) completed follow-up surveys. VA enterprise-level MHV log-in data on all participants for the full 18-month time frame became available to the investigators in 2015.

    The study cohort was comprised mostly of men (228/269, 84.8%) who were white (223/270, 82.6%) and over the age of 50 years (184/270, 68.1%; Multimedia Appendix 1). Representation from women (41/269, 15.2%) was somewhat greater than the VA population of approximately 12% [26]. Fewer than 1 in 5 (46/258, 17.8%) participants had a high school education or less, approximately half completed some college (127/256, 49.2%), and one-third were college graduates (85/258, 32.9%). Health literacy screening found 98.1% (261/266) of participants in the adequate category. A total of 48.7% (128/263) of participants stated their health status as fair or poor, with only 15.0% (40/266) reporting not having a chronic condition or disability; 38.7% (103/266) resided more than a one-hour drive from the nearest VA facility.

    Almost all study participants (260/268, 97.0%) reported going online at least occasionally, most commonly at home (248/268, 92.5%). A total of 32.3% (86/266) rated their computer ability as intermediate and 57.5% (153/266) rated it as advanced. Just over half (144/266, 54.1%) of the respondents indicated that they had used public Wi-Fi at an airport, coffee shop, or restaurant, while 41.9% (111/265) went online using a mobile device such as an iPhone or other mobile phone or tablet. Most respondents indicated that they had searched online to find information (256/269, 95.2%), to map a specific location (254/268, 94.8%), to shop or purchase a product (219/269, 81.4%), and for banking to pay bills (203/267, 76.0%).

    Short-term use of the portal was examined by analyzing the number of successful log-ins for each study participant in the 6 months following study enrollment. Long-term usage examined the total portal log-ins over 18 months. Log-ins were also measured during the time interval between 6 and 18 months following study enrollment. The distribution of log-ins for each time period is shown in Figure 1 and Figure 2, respectively (total log-ins are capped at 78 for 6-months, and at 234 for 18-months). At 6 months, the mean number of log-ins was 14.1 (standard deviation [SD] 22.7), with a median of 7, a minimum of 0, and a maximum of 169 log-ins; 75.9% (205/270) of participants had 17 or fewer log-ins. At 18 months, the mean number of log-ins was 34.8 (SD 48.7), with a median of 17, a minimum of 0, and a maximum of 407 log-ins; 75.2% (203/270) of participants had 49 or fewer log-ins.

    Portal usage at 6 months, as measured by successful log-ins, is shown in Table 1. Log-in count is shown in four categories: 0 or 1 log-ins; 2 to 5 log-ins, or approximately less than once per month; 6 to 11 log-ins, or once/twice per month; and 12 or more log-ins, or over twice per month. Six months after having full access to all portal functions, approximately one-third of patients logged in less than once per month, and half logged in approximately monthly or more. Just under 16% (43/270) of patients had never logged in over 6 months.

    Six-month portal usage demonstrated some variation across demographic and health-related subgroups; however, there were no statistically significant differences in log-in behavior by age, gender, education level, marital status, race/ethnicity, or distance to a VA facility. Likewise, self-reported health status, having a specific chronic condition, smoking status, or previously obtaining copies of health records were not significantly associated with variation in usage. The baseline PAM was not predictive of future short-term patient portal usage.

    Table 2 shows the relationship between portal usage and participants’ reports of their technology access and Internet-related factors. Expected differences were found in the use of the portal, with significantly greater usage among those having Internet at home, having a high-speed broadband connection, and greater self-reported ability to use the Internet. The 42.5% (113/266) of participants who reported novice or intermediate abilities using the Internet used the portal less often compared to subjects reporting advanced Internet skills. More frequent log-ins were observed for regular email users, and those who were online more often, or who more frequently searched for health information. Our composite Internet Use measure found that while 80.3% (217/270) of the cohort fell into the high use category, 43.4% (23/53) of participants in the low use category never logged in or did so only once.

    Patient concerns about online privacy showed no significant relationship to portal usage. Higher numbers of portal log-ins were associated with learning about MHV by reading promotional materials or from recommendations by a clinician. Portal use was also higher in participants who had visited MHV prior to completing the in-person verification step for a full access account. Participants’ views of anticipated use of specific portal functions appeared to play little role in future portal usage.

    Portal usage during the 18-month period is shown in Table 3. By 18 months, never-users decreased to 9%, showing a delay in first use for some patients. Similar to the 6-month results, no significant associations were found between usage and demographics, self-reported health status, or PAM score. When assessing participant self-reports of comorbid conditions, there was a trend showing higher usage associated with having more chronic conditions, although this trend was not statistically significant. Overall use of the Internet was strongly associated with long-term portal use, similar to short-term use (data not shown).

    When examining portal log-in frequency during the time interval between 6 and 18 months, as shown in Table 4, the trend between chronic conditions and log-ins was no longer observed.

    Log-ins during the first 6 months after enrollment were compared to portal activity during the time interval between 6 and 18 months. Shown in Table 5, patients who rarely used the portal most commonly remained in that category. Similarly, greater usage during the first 6 months led to similar log-in behavior during the later time interval.

    Many participants experienced an unsuccessful log-in during the study period; this occurred when a user entered an invalid username or password. At 6 months, a total of 169 subjects (169/270, 62.6%) had at least one unsuccessful log-in with a mean of 3.5 unsuccessful log-in attempts (SD 5.9). At 18 months, 214 subjects (217/270, 79.3%) experienced at least one unsuccessful log-in with a mean of 7.6 (SD 11.2) unsuccessful attempts.

    Figure 1. Distribution of the number of patient logins 6 months following full portal access. Total logins are capped, with participants having 78 or more logins shown at the highest count.
    View this figure
    Table 1. Association of demographics and health factors with portal usage 6 months after full access.
    View this table
    Figure 2. Distribution of the number of patient logins 18 months following full portal access. Total logins are capped, with participants having 234 or more logins shown at the highest count.
    View this figure
    Table 2. Association of Internet access and online activity with portal usage 6 months after full access.
    View this table
    Table 3. Association of demographics and health factors with portal usage 18 months after full access.
    View this table
    Table 4. Association of demographics and health factors with portal between 6 months and 18 months after full access.
    View this table
    Table 5. Relationship of portal usage in the 6 months after enrollment and from 6 to 18 months.
    View this table

    Discussion

    Among this VA cohort who took active steps to enroll in, and gain access to, patient portal functions, short-term and long-term portal usage were significantly associated with having broadband Internet at home, high self-rated ability to use the Internet, and overall online behavior. Access to broadband Internet has emerged as a social determinant of health [27], defined as, “a condition in which people are born, grow, live, work, and age, and which shapes their health status” [28]. As virtual health care becomes more commonplace, affordable broadband Internet and devices, and digital know-how, will be needed to ensure equity in care services [29].

    In 2009, Kahn et al [30] identified Internet access and digital skills as being vital for the success of PHRs, stating, “if these are not made policy priorities, PHRs risk becoming a tool that is limited to groups of people who are already linked to the Internet with high health literacy and computer skills.” Our study findings indicate that ready access to the Internet and digital skills, often referred to as digital inclusion [31] and broader in scope than Internet access, appears to be a social determinant for exposure to patient portal services. These results expand upon prior research showing that portal adoption is associated with digital competency and Internet access [32], with usage dependent upon user skills and user-centered design of digital tools [33,34].

    We expected to find that patient activation was related to more frequent portal use, but this was not the case. Limited studies examining this relationship have produced varied findings. One study found modest increases in PAM scores in portal users compared to a control group [35]. Others found no significant change in PAM scores among patients with chronic illness who were given access to a portal [36], and no association between PAM and portal log-in frequency [37]. Given our findings correlating portal use and digital access and skills, a more complex relationship likely exists between patient activation and online behavior. Larger prospective studies capturing patients’ digital environments and behaviors could offer greater insights into a patient’s context and clinical trajectory that are certain to play a role in portal usage (eg, a new diagnosis or worsening of a health condition). It is also important to characterize stops along the way to sustained use of digital tools [38]. Each step on the journey of patient portal engagement presents barriers and drivers to continued use [39]. Researchers should specify their targets, as there are several, including: patients routinely given a code to establish an account (which can occur with or without Internet access); patients who activate their accounts or log in at least once to test it; and ultimately, patients with sustained portal usage.

    Expanding health care to virtual channels may create greater care disparities among those without affordable and reliable access to the Internet or digital devices. A focus on mutable factors that can positively impact the reach and meaningful use of portals is essential. Marketing and promotion is important, since patients may not be aware of portals, or do not perceive their value [40]. Kaiser Permanente has made a digital strategy an operational goal, and has achieved a high rate of portal engagement and use, by routinely enrolling all members and making benefits clear [41]. Clinicians also play a key role in promoting portals and elevating their significance (eg, by offering the option of secure email and explaining its appropriate use) [42].

    Once online, patients will need to easily navigate portals and be able to intuitively use the tools to meet their needs, which is a necessity for sustained usage [43-45]. As consumer trends show shifts in digital devices toward mobile options, patient-facing tools need to be device agnostic. In the United States, low-income adults in particular are increasingly accessing the Internet only via mobile devices [46]. Studies in safety net and senior populations show that many people in these groups have mobile phones, and smartphones are increasingly substituting for computers [47].

    Finally, user-centered designs that optimize portal usability will amplify their use [48]. Ease of entry is critical to patient satisfaction. Even the presumably simple task of securely logging-in can challenge users. Indeed, we found that many participants experienced unsuccessful log-ins. Usability challenges have been found in the VA portal, including complex password requirements for log-in [49]. Balancing security with usability is critical; users facing difficulty logging into a portal may abandon efforts altogether.

    The capability to effectively use the Internet is vital for health care, as well as for education and employment. Society’s digital revolution is evolving faster than our ability to accurately measure and demonstrate digital competence across all populations [50,51]. At a minimum, health care and public health stakeholders should align across industry sectors, partnering for economic development and investment to improve community broadband and digital literacy, in an effort to achieve digital inclusion.

    Limitations

    There are limitations to our study. Patients were recruited at a single VA facility, and may not represent a general population of patients or those enrolling for the portal. Participants may be more computer-savvy than the general portal user population. Many participants visited the MHV website before the study (VA patients who register but do not complete identify proofing can refill medications using a prescription number). The study setting in a busy MHV office precluded capturing data on all patients informed about the study. However, such issues could potentially underestimate challenges that users faced using the portal. Second, measuring the portal served as a proxy for usage and benefit. While standardized metrics for capturing patient usage of digital tools have not been established, common measures include initial enrollment, log-in frequency, and using specific functions [34]. Measuring total log-ins during 6-month and 18-month intervals is not ideal; however, repeated log-ins over time signals user value. Third, our health literacy assessment found virtually all patients at the highest level of the S-TOFHLA; the acceptability of this instrument has since been questioned [52]. Finally, study subjects may have experienced inconsistent connections to the Internet over the study, complicating the measurements of associations between digital inclusion and portal use.

    Conclusion

    The ultimate impact of patient portals will rest on their ability to reach across populations and have real-world effects on self-care and outcomes. Realizing potential benefits will require not just initial adoption but also sustained portal usage. Strategies and novel methods to enhance comfort with digital devices and increase Internet skills, along with affordable access to broadband and wireless connections, are required ingredients as the health care community offers an increasing array of online tools and services. There are important relationships between digital inclusion and the use and benefits of virtual care tools.

    Conflicts of Interest

    Judith Hibbard is a consultant to, and equity stakeholder in, Insignia Health. All other authors declare no conflicts of interest.

    Multimedia Appendix 1

    Study cohort self-reported demographics, conditions, patient activation, and Internet access.

    PDF File (Adobe PDF File), 24KB

    References

    1. Centers for Medicare and Medicaid. EHR Incentive Programs: 2015 through 2017 (Modified Stage 2) Overview. 2017.   URL: https:/​/www.​cms.gov/​Regulations-and-Guidance/​Legislation/​EHRIncentivePrograms/​Downloads/​2015_EHR2015_2017.​pdf [accessed 2017-04-20] [WebCite Cache]
    2. Fox S. Pew Internet & American Life Project. 2011 Feb 01. Health topics: 80% of Internet users look for health information online   URL: http://www.pewinternet.org/files/old-media/Files/Reports/2011/PIP_Health_Topics.pdf [accessed 2017-09-23] [WebCite Cache]
    3. Ahern D, Woods S, Lightowler M, Finley S, Houston T. Promise of and potential for patient-facing technologies to enable meaningful use. Am J Prev Med 2011 May;40(5 Suppl 2):S162-S172. [CrossRef] [Medline]
    4. Gruman J. Making health information technology sing for people with chronic conditions. Am J Prev Med 2011 May;40(5 Suppl 2):S238-S240. [CrossRef] [Medline]
    5. Perrin A, Duggan M. Pew Research Center. 2015 Jun 26. Americans' Internet Access: 2000-2015   URL: http:/​/assets.​pewresearch.org/​wp-content/​uploads/​sites/​14/​2015/​06/​2015-06-26_internet-usage-across-demographics-discover_FINAL.​pdf [accessed 2017-09-23] [WebCite Cache]
    6. Gordon N, Hornbrook M. Differences in access to and preferences for using patient portals and other eHealth technologies based on race, ethnicity and age: a database and survey study of seniors in a large health plan. J Med Internet Res 2016;18(3):a. [CrossRef]
    7. Yamin CK, Emani S, Williams DH, Lipsitz SR, Karson AS, Wald JS, et al. The digital divide in adoption and use of a personal health record. Arch Intern Med 2011 Mar 28;171(6):568-574. [CrossRef] [Medline]
    8. Smith SG, O'Conor R, Aitken W, Curtis LM, Wolf MS, Goel MS. Disparities in registration and use of an online patient portal among older adults: findings from the LitCog cohort. J Am Med Inform Assoc 2015 Jul;22(4):888-895 [FREE Full text] [CrossRef] [Medline]
    9. Ancker JS, Hafeez B, Kaushal R. Socioeconomic disparities in adoption of personal health records over time. Am J Manag Care 2016 Aug;22(8):539-540 [FREE Full text] [Medline]
    10. Gardina T, Menon S, Parrish D, Sittig D, Singh H. Patient access to medical records and healthcare outcomes: a systematic review. J Am Inform Assoc 2014;21:a. [CrossRef]
    11. Goldzweig C, Orshansky G, Paige N, Towfigh A, Haggstrom D, Miake-Lye I, et al. Electronic patient portals: evidence on health outcomes, satisfaction, efficiency, and attitudes: a systematic review. Ann Intern Med 2013 Nov 19;159(10):677-687. [CrossRef] [Medline]
    12. Wright E, Darer J, Tang X, Thompson J, Tusing L, Fossa A, et al. Sharing physician notes through an electronic portal is associated with improved medication adherence: quasi-experimental study. J Med Internet Res 2015 Oct 08;17(10):e226 [FREE Full text] [CrossRef] [Medline]
    13. Woods SS, Schwartz E, Tuepker A, Press NA, Nazi KM, Turvey CL, et al. Patient experiences with full electronic access to health records and clinical notes through the My HealtheVet Personal Health Record Pilot: qualitative study. J Med Internet Res 2013 Mar 27;15(3):e65 [FREE Full text] [CrossRef] [Medline]
    14. Nazi K, Turvey C, Klein D, Hogan T, Woods S. VA OpenNotes: exploring the experiences of early patient adopters with access to clinical notes. J Am Med Inform Assoc 2015 Mar;22(2):380-389. [CrossRef] [Medline]
    15. McInnes D, Shimada S, Midboe A, Nazi K, Zhao S, Wu J, et al. Patient use of electronic prescription refill and secure messaging and its association with undetectable HIV viral load: a retrospective cohort study. J Med Internet Res 2017 Feb 15;19(2):e34 [FREE Full text] [CrossRef] [Medline]
    16. Ahmed S, Ernst P, Bartlett SJ, Valois MF, Zaihra T, Paré G, et al. The effectiveness of web-based asthma self-management system, My Asthma Portal (MAP): a pilot randomized controlled trial. J Med Internet Res 2016 Dec 01;18(12):e313 [FREE Full text] [CrossRef] [Medline]
    17. Shimada SL, Allison JJ, Rosen AK, Feng H, Houston TK. Sustained use of patient portal features and improvements in diabetes physiological measures. J Med Internet Res 2016 Jul 01;18(7):e179 [FREE Full text] [CrossRef] [Medline]
    18. Fox S, Duggan M. Pew Research Center. 2013 Nov 26. The diagnosis difference   URL: http://www.pewinternet.org/files/old-media//Files/Reports/2013/PewResearch_DiagnosisDifference.pdf [accessed 2017-09-23] [WebCite Cache]
    19. Houston TK, Volkman JE, Feng H, Nazi KM, Shimada SL, Fox S. Veteran Internet use and engagement with health information online. Mil Med 2013 Apr;178(4):394-400. [CrossRef] [Medline]
    20. Lyles C, Sarkar U, Schillinger D, Ralston J, Allen J, Nguyen R, et al. Refilling medications through an online patient portal: consistent improvements in adherence across racial/ethnic groups. J Am Med Inform Assoc 2016 Apr;23(e1):e28-e33 [FREE Full text] [CrossRef] [Medline]
    21. Ralston JD, Silverberg MJ, Grothaus L, Leyden WA, Ross T, Stewart C, et al. Use of web-based shared medical records among patients with HIV. Am J Manag Care 2013 Apr 01;19(4):e114-e124 [FREE Full text] [Medline]
    22. U.S. Department of Veterans Affairs. 2017. My HealtheVet   URL: https://www.va.gov/vetdata/ [accessed 2017-04-20] [WebCite Cache]
    23. Baker DW, Williams MV, Parker RM, Gazmararian JA, Nurss J. Development of a brief test to measure functional health literacy. Patient Educ Couns 1999 Sep;38(1):33-42. [Medline]
    24. Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004 Aug;39(4 Pt 1):1005-1026 [FREE Full text] [CrossRef] [Medline]
    25. StataCorp. Stata Statistical Software: Release 14. College Station, TX; 2015.   URL: https://www.stata.com/support/faqs/resources/citing-software-documentation-faqs/ [accessed 2017-09-23] [WebCite Cache]
    26. U.S. Department of Veterans Affairs. National Center for Veterans Analysis and Statistics. 2017.   URL: https://www.va.gov/vetdata/ [accessed 2017-04-20] [WebCite Cache]
    27. Perzynski AT, Roach MJ, Shick S, Callahan B, Gunzler D, Cebul R, et al. Patient portals and broadband internet inequality. J Am Med Inform Assoc 2017 Mar 23:927-932. [CrossRef] [Medline]
    28. Chapman AR. The social determinants of health, health equity, and human rights. Health Hum Rights 2010 Dec 15;12(2):17-30. [Medline]
    29. Graetz I, Gordon N, Fung V, Hamity C, Reed ME. The digital divide and patient portals: Internet access explained differences in patient portal use for secure messaging by age, race and income. Med Care 2016 Aug;54(8):772-779. [CrossRef] [Medline]
    30. Kahn JS, Aulakh V, Bosworth A. What it takes: characteristics of the ideal personal health record. Health Aff (Millwood) 2009;28(2):369-376 [FREE Full text] [CrossRef] [Medline]
    31. Institute of Museum and Library Services, University of Washington Technology & Social Change Group, International City/County Management Association. Building Digitally Inclusive Communities: A Guide to the Proposed Framework. 2017.   URL: https://www.imls.gov/assets/1/AssetManager/DIC-FrameworkGuide.pdf [accessed 2017-04-20] [WebCite Cache]
    32. Tieu L, Schillinger D, Sarkar U, Hoskote M, Hahn K, Ratanawongsa N, et al. Online patient websites for electronic health record access among vulnerable populations: portals to nowhere? J Am Med Inform Assoc 2017 Apr 01;24(e1):e47-e54. [CrossRef] [Medline]
    33. Irizarry T, DeVito Dabbs A, Curran C. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015 Jun 23;17(6):e148 [FREE Full text] [CrossRef] [Medline]
    34. Jimison H, Gorman P, Woods S, Nygren P, Walker M, Norris S, et al. Barriers and drivers of health information technology use for the elderly, chronically ill, and underserved. Evid Rep Technol Assess (Full Rep) 2008 Nov(175):1-1422. [Medline]
    35. Nagykaldi Z, Aspy C, Chou A, Mold J. Impact of a Wellness Portal on the delivery of patient-centered preventive care. J Am Board Fam Med 2012;25(2):158-167 [FREE Full text] [CrossRef] [Medline]
    36. Riippa I, Linna M, Rönkkö I. A patient portal with electronic messaging: controlled before-and-after study. J Med Internet Res 2015 Nov 09;17(11):e250 [FREE Full text] [CrossRef] [Medline]
    37. Anker J, Osorio S, Cheriff A, Cole C, Silver M, Kaushal R. Patient activation and use of an electronic patient portal. Inform Health Soc Care 2015;40(3):254-266. [CrossRef]
    38. Otte-Trojel T, de Bont A, Rundall T, van de Klundert J. How outcomes are achieved through patient portals: a realist review. J Am Med Inform Assoc 2014;21(4):751-757 [FREE Full text] [CrossRef] [Medline]
    39. Ancker JS, Barrón Y, Rockoff ML, Hauser D, Pichardo M, Szerencsy A, et al. Use of an electronic patient portal among disadvantaged populations. J Gen Intern Med 2011 Oct;26(10):1117-1123 [FREE Full text] [CrossRef] [Medline]
    40. California Healthcare Foundation. Patient Portals in the Safety Net: Phase 3 Optimization. 2013 Apr.   URL: http://www.chcf.org/~/media/MEDIA%20LIBRARY%20Files/PDF/PDF%20P/PDF%20PatientPortalOptimization.pdf [accessed 2017-04-20] [WebCite Cache]
    41. Garrido T, Raymond B, Wheatley B. Health Affairs Blog. 2016 Apr 07. Lessons from more than a decade in patient portals   URL: http://healthaffairs.org/blog/2016/04/07/lessons-from-more-than-a-decade-in-patient-portals/ [accessed 2017-04-20] [WebCite Cache]
    42. Crotty BH, Tamrat Y, Mostaghimi A, Safran C, Landon BE. Patient-to-physician messaging: volume nearly tripled as more patients joined system, but per capita rate plateaued. Health Aff (Millwood) 2014 Oct;33(10):1817-1822 [FREE Full text] [CrossRef] [Medline]
    43. Czaja SJ, Zarcadoolas C, Vaughon WL, Lee CC, Rockoff ML, Levy J. The usability of electronic personal health record systems for an underserved adult population. Hum Factors 2015 May;57(3):491-506 [FREE Full text] [CrossRef] [Medline]
    44. Crotty BH, Slack WV. Designing online health services for patients. Isr J Health Policy Res 2016;5:22 [FREE Full text] [CrossRef] [Medline]
    45. Woods SS, Evans NC, Frisbee KL. Integrating patient voices into health information for self-care and patient-clinician partnerships: Veterans Affairs design recommendations for patient-generated data applications. J Am Med Inform Assoc 2016 May;23(3):491-495. [CrossRef] [Medline]
    46. Pew Research Center. 2015 Dec 21. Home Broadband 2015   URL: http://assets.pewresearch.org/wp-content/uploads/sites/14/2015/12/Broadband-adoption-full.pdf [accessed 2017-09-23] [WebCite Cache]
    47. Pew Research Center. 2015 Apr 01. U.S. Smartphone Use in 2015   URL: http://www.pewinternet.org/files/2015/03/PI_Smartphones_0401151.pdf [accessed 2017-09-23] [WebCite Cache]
    48. Council of Economic Advisors Issue Brief. 2015 Jul. Mapping the digital divide, July 2015   URL: https://obamawhitehouse.archives.gov/sites/default/files/wh_digital_divide_issue_brief.pdf [accessed 2017-04-20] [WebCite Cache]
    49. Haggstrom DA, Saleem J, Russ A, Jones J, Russell S, Chumbler N. Lessons learned from usability testing of the VA's personal health record. J Am Med Inform Assoc 2011 Dec;18 Suppl 1:i13-i17 [FREE Full text] [CrossRef] [Medline]
    50. van der Vaart R, Drossaert C. Development of the digital health literacy instrument: measuring a broad spectrum of Health 1.0 and Health 2.0 skills. J Med Internet Res 2017 Jan 24;19(1):e27 [FREE Full text] [CrossRef] [Medline]
    51. Boot W, Charness N, Czaja S, Sharit J, Rogers WA, Fisk AD, et al. Computer proficiency questionnaire: assessing low and high computer proficient seniors. Gerontologist 2015 Jun;55(3):404-411 [FREE Full text] [CrossRef] [Medline]
    52. Altin SV, Finke I, Kautz-Freimuth S, Stock S. The evolution of health literacy assessment tools: a systematic review. BMC Public Health 2014 Nov 24;14:1207 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    MHV: My HealtheVet
    PAM: Patient Activation Measure
    SD: standard deviation
    S-TOFHLA: Short Form Functional Health Literacy Assessment
    VA: Veterans Affairs


    Edited by G Eysenbach; submitted 20.04.17; peer-reviewed by K McInnes, C Zarcadoolas; comments to author 18.05.17; revised version received 13.07.17; accepted 31.07.17; published 17.10.17

    ©Susan S Woods, Christopher W Forsberg, Erin C Schwartz, Kim M Nazi, Judith H Hibbard, Thomas K Houston, Martha Gerrity. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.10.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.