Health care systems have entered a new era focused on patient engagement [1-3] described by enthusiasts as “the holy grail of health care” [4] and the “blockbuster drug of the century” [5]. Patient engagement strategies are designed to empower patients to play a more active role in their health care and make informed decisions, improve the patient experience, increase patient satisfaction, and achieve better health outcomes. Patient portals linked to electronic health records (EHRs) are recognized as a promising multifaceted tool to help achieve these patient engagement goals [6-9]. However, the adoption and sustained use of portals has generally fallen short of initial optimism [10-13] even in light of the significant growth in EHRs and tethered patient portals incentivized by Meaningful Use [14]. Positive benefits of portal use have been demonstrated [15-19], and the OpenNotes movement [20] has promoted patient engagement through health records transparency by enabling patient access to provider notes. Evidence indicates that such access improves communication and trust, patient safety, and, potentially, patient outcomes [21-25]. Two large integrated health care systems that launched tethered patient portals in 2003 with significant patient adoption and sustained use are Kaiser Permanente (KP) and the Department of Veterans Affairs (VA). KP’s portal, My Health Manager, is used by more than 5 million members, representing about 70% of adult KP members [26]. VA’s patient portal, My HealtheVet, has more than 4 million registered users (69% of VA patients receiving health care services in FY 2017 [wherein “FY” means “fiscal year”]), with 2.5 million authenticated Premium accounts (42% of VA patients receiving health care services) required for access to all portal features [27]. To better understand what elements are driving this adoption and sustained use, an implementation case study approach is warranted. One of the implementation strategies that are critical to foster the adoption and sustained use of patient portals is an ongoing focus on user-centered design (UCD). This is often accomplished as part of periodic research studies; however, more timely and agile methods are needed to design and evaluate patient portals.

UCD is a design philosophy and evaluation process that focuses on the end user’s characteristics, needs, preferences, and limitations throughout the design process and development lifecycle [28]. The emphasis of UCD is on understanding the end users and their tasks and goals and optimizing the product to enable the users to fulfill these, rather than requiring users to adapt to the designer’s preferences [29]. UCD of eHealth applications, such as patient portals, necessitates ongoing assessment of user characteristics and preferences and incorporation of assessment insights into ongoing portal development and enhancements. This process includes focusing on what features are considered to be most essential by users [30]. Published compilations of implementation strategies have called for the further development of processes like UCD as a means to obtain and use patient or consumer feedback to support the adoption of innovations and practice change efforts in health care and other settings [31].

VA has used various methods over the last decade to achieve UCD for My HealtheVet; among them, the principal method has been a continuous, voluntary, and anonymous survey of end users. As a complement to periodic formal research studies [32,33], this ongoing assessment offers the advantage of rapid continuous feedback, which is part of a cyclical process for improvement that entails understanding users, eliciting their input, identifying changes or future design implications, deploying enhancements, and then obtaining feedback to evaluate these enhancements. This method enables VA to obtain ongoing direct feedback from veterans, which can then be leveraged to improve the patient experience.

VA is the largest integrated health care system in the United States and has been a pioneer in enabling patients to access and download their VA medical record data using the Blue Button feature [34,35]. This includes OpenNotes, which are known in My HealtheVet as VA Notes and contain both clinical and mental health providers’ notes [36]. The My HealtheVet patient portal [37] is tethered to the VA EHR and provides a suite of Web-based tools. Veterans self-register to create a basic account and can then self-enter information into their personal health record and access health education resources. VA patients who are matched by the system via the Master Veteran Index are automatically upgraded to an Advanced account and can request VA prescription refills. Patients who complete a one-time process of identity authentication (in person or Web-based) are upgraded to a Premium account and can then access all portal features, including access to health record information and Secure Messaging with VA health care professionals.

Use of My HealtheVet continues to grow. In fiscal year (FY)2017, portal user activity demonstrated significant increases compared with that in FY2016, including a 20.7% increase in Web-based prescription refills, a 33.9% increase in Secure Messaging exchanges between VA patients and their health care team, and a 38.7% increase in use of the VA Blue Button feature [27].

In this paper, we examine one of the implementation strategies that VA has used to foster adoption and sustained use of its patient portal over the last decade: an ongoing focus on UCD. This includes iterative use of survey and operational data with user interface redesign to meet the needs and preferences of veteran users. We describe the organization’s implementation strategy for agile UCD and present unique 10-year data on user adoption, characteristics, and utilization to demonstrate trends toward greater patient engagement and satisfaction. Following an initial analysis of portal users and their preferences in 2007 [38], we compared the characteristics of patient portal users one decade later and used a case study approach to present several examples of how user preferences and continuous feedback have informed the evolution of VA’s patient portal.

We first present a recent summary of user demographics and characteristics and patterns of portal use and relevant comparisons to previous data. Following our case study approach, we then provide selected examples from the My HealtheVet evaluation program to illustrate how different assessments that capture the voice of the customer have directly informed the evolution of the portal and the addition of new functionality. For FY2008 (October 1, 2007-September 30, 2008), of the surveys presented to site visitors, 17.1% (100,069/585,039) were completed. For FY2017 (October 1, 2016-September 30, 2017), of the surveys presented to site visitors, 68.9% (100,555/146,023) were completed. As completion rates increased over the last decade, the sampling rate was reduced in FY2010 from 13% to 4% in order to minimize respondent burden.

Table 1 provides a comparison of user demographics and characteristics for all survey respondents in FY2017 and FY2008. In FY2017, 97% (97,538/100,555) of respondents were veterans compared with 93% (93,064/100,069) in FY2008. Respondents reported having completed higher levels of education, with 40% (39,990/99,974) being college graduates, completing some postgraduate school, or having a graduate or professional degree in FY2017 compared with 34% (732/2152) in FY2008. The proportion of male respondents increased slightly to 93% (90,507/97,319) in FY2017. In FY2017, respondents were generally older, with 64% (59,819/93,467) in the age range of 60-74 years compared with 47% (14,563/30,984) in FY2008; furthermore, 17% (15,889/93,467) of respondents in FY2017 were older than 75 years. This shift in age is also shown in Figure 1.

While 60% (60,042/100,069) of users in FY2008 reported their period of military service as the Vietnam War, this increased to 67% (67,372/100,555) of users in FY2017. Fewer users self-reported their internet ability as advanced in FY2017 (32,235/53,725, 60%) than in FY2008 (37,848/55,658, 68%), whereas more users reported it as intermediate in FY2017 (19,341/53,725, 36%) than in FY2008 (16,141/55,658, 29%). A greater proportion of respondents reported better health in FY2017, with 34% (33,323/98,007) reporting fair or poor health in FY2017 compared with 39% (15,723/40,315) in FY2008. Although the FY2008 survey did not ask users about health conditions, responses in FY2017 revealed a high prevalence of chronic conditions, including high blood pressure (15,045/22,795, 66%), high cholesterol (14,133/22,795, 62%), arthritis (13,677/22,795, 60%), chronic pain (10,714/22,795, 47%), diabetes (8434/22,795, 37%), stomach or gastrointestinal problems (8434/22,795, 37%), and heart problems (8434/22,795, 37%).

Table 2 provides a summary of survey respondents’ self-reported portal access patterns and usage. While the proportion of survey respondents who use VA health care services remained the same from FY2008 to FY2017 (96%), in FY2017 50% (47,066/94,132) of respondents stated that they also use a community non-VA provider. When asked about travel time to the nearest VA facility, 32% respondents reported it to be less than 30 minutes in both FY2008 and FY2017; however, a greater proportion noted fewer minutes of travel time in FY2017 than in FY2008. For example, 37% (19,902/53,788) respondents in FY2008 reported a travel time greater than 60 minutes to the nearest VA facility compared with 24% (12,075/50,313) in FY2017. The proportion of respondents who have a Premium account, offering them access to all portal services, increased significantly from 60% (56,884/94,806) in FY2008 to 77% (73,001/94,806) in FY2017. While a greater number of respondents were first time users in FY2008 (12,074/100,617, 12%) than in FY2017 (4022/100,555, 4%), respondents reported using the portal more frequently in FY2017, with 46% (45,255/100,555) using it about once a month and 29% (29,161/100,555) using it about once a week. When asked about the length of use in FY2017, 63% (63,349/100,5455) respondents reported having used My HealtheVet for more than 2 years.

In keeping with the UCD process, VA has used direct veteran feedback about preferences obtained via the CXA survey to shape the identification and prioritization of portal improvements. In this section, we describe how different types of user feedback have directly informed enhancements to the core components of the system including available features, interface design, content, policy, and prospective functional design of a new feature.

Analysis of CXA data over the course of the last decade has been an integral part of the recent My HealtheVet website redesign initiative by enabling a deeper understanding of the end users and their tasks and goals, in keeping with UCD principles. As shown in Table 5, while 75% (75,241/100,617), 24% (23,923/100,617), and 18% (17,899/100,617) users in FY2008 accessed the portal to request a prescription refill, view their medication history, and look up information about a medication, respectively, user goals and tasks in FY2017 have shifted and expanded. Although prescription refill requests remained a predominant task (53,193/100,555, 53%), users also accessed the portal to view their VA Appointments (38,664/100,555, 38%), communicate with their health care team using Secure Messaging (28,952/100,555, 29%), track the delivery status of their medication refills (23,884/100,555, 24%), view their lab or test results (19,382/100,555, 19%), and access their VA health records (11,966/100,555, 12%). An important goal of the culminating website redesign was to improve navigation and usability for these specific core features, and the overall customer satisfaction index score was used as a performance indicator.

Historical customer satisfaction trends are shown in Figure 3. From October 2007 to October 2015, the aggregate average CXA score was 74, based on 945,480 completed surveys. The average for the 12 months that followed was stable at 76 (N=139,934). While multiple factors impacted customer satisfaction over the last decade, including a period of system performance issues in 2014 that was resolved by improving system architecture, the overall trend toward greater customer satisfaction is evident.

In October 2016, as part of an incremental website redesign, a dashboard was added to the portal home page to enhance user access to the core features (Figure 4).

As anticipated, the introduction of changes to the website resulted in an initial decrease in satisfaction (72), followed by satisfaction recovery (75), and subsequent increase to a new high of 79 (Figure 5). A similar pattern was observed with the deployment of additional website redesign changes in September 2017. Satisfaction initially decreased (77), but then recovered to previous levels (79). Satisfaction continued to increase in January 2018 (80).

The literature on adoption and use of patient portals highlights the need for health care organizations to employ UCD approaches to ensure that portals align with end users’ characteristics, needs, preferences, and goals, and, ultimately, help advance portal implementation. In its commitment to UCD, one method that VA has used to accomplish the above is a continuous survey to elicit a direct feedback from a random sample of veterans who use VA’s patient portal, My HealtheVet. In combination with other methods, such as targeted research studies, the CXA survey has enabled a deeper understanding of portal users and directly informed changes in portal features, functions, policies, and processes. By incorporating the results of this systematic evaluation of the user experience into the portal redesign, VA aims to continue to enhance the ability of My HealtheVet to engage and activate veterans in managing their health.

This study compared the characteristics and behaviors of users during the early period of patient portal implementation, 2008, with that of later adopters. This provided a trajectory of how portal use has evolved over a decade. Many aspects remained stable, while others showed clear trends toward portal adoption by populations believed less likely to use patient-facing health technologies. While only 13.4% (89,780/670,000) portal users in FY2008 were VA patients with a Premium account, by FY2017, this increased to 62.5% (2.5 million/4 million) users. Despite early assumptions about older users not adopting and using patient portals [46,47], leading to a gray digital divide [48], the VA experience reveals an increasingly elderly population of users. Within the veteran population, research has shown that VA patients tend to be older and more socioeconomically disadvantaged than veterans who do not rely on VA for care [49]. Although the survey indicates that the majority of users have one or more chronic health conditions and access the portal with increasing frequency, the survey results also suggest a trend toward those with less internet ability and better health also accessing the patient portal. This trend may be a result of the portal expanding the types of transactional services that users find convenient, based on direct veteran input. It also suggests that the portal is engaging a broader segment of the veteran population. Although the proportion of female veterans responding to the survey decreased slightly in FY2017 (from 9% to 7%), the overall population of female veterans was estimated to be 9.4% in 2015. However, only 22.4% used VA health care services [50], which is a key driver for accessing the patient portal. Portal users in FY2017 also tended to have completed higher levels of education than those in FY2008. This may be reflective of changes in the veteran population overall, with the enhanced provision of educational support programs for separating service members. Given that half of the survey respondents in FY2017 reported that they also use community non-VA care providers, VA will need to continue to develop tools that enable effective information sharing across settings of care. Portal functions that support consumer-mediated health information exchange are currently in early field testing [51]. These patient portal user trends align with similar trends for the VA patient population overall in terms of gender (91% male), age (median age of male VA patients, 64 years), and increasing use of VA education benefits [52].

Based on user self-report about goals and tasks, a significant redesign of the website was undertaken to enhance navigation to the features aligned with the most common user tasks and to decrease the number of steps to accomplish these. After an initial period of satisfaction decline, anticipated due to the phenomenon of change aversion [53], the satisfaction index recovered and increased. Once users adjusted to the change, they were more satisfied with the new design as measured using the CXA satisfaction index. Looking ahead, there are additional improvements and enhancements that will be important to address.

It is important to note that the results of the CXA survey reflect the characteristics and perspectives of a random sample of portal users who are invited and opt to participate in the survey and may not be fully generalizable to the larger population. More broadly, the respondent sample represents patient portal users; therefore, other methods are also needed to elicit input from veterans who are not portal users to understand their characteristics and preferences and identify barriers that may exist to system access and use. VA is in the process of adding questions to its patient experience survey, administered to veterans who had a recent medical encounter, to help fill this gap, and ongoing research about veteran preferences for digital tools and services provides complementary insights [32,33]. There may also be data that were not collected in the survey that could be important. Since the survey is anonymous, there is no opportunity to follow up with respondents for more information or clarification. Despite limitations inherent to an anonymous survey, it has the benefit of enabling a continuous flow of direct feedback. While the findings from our case study may not be fully generalizable to other patient populations, the principle of using agile approaches to employ UCD has potential to be a promising implementation strategy for other health care organizations.

By leveraging UCD principles, VA has continued to enhance its patient portal and supported its continued implementation, achieving significant growth in adoption and use over the last decade. While quantitative and qualitative research studies are an important component of patient portal evaluation, more agile methods are also needed to complement formal research efforts. As illustrated through this case study, we have found the ongoing administration of a continuous voluntary Web-based survey as an efficient and effective way to capture veteran’s voices about who they are, how they use the patient portal, what improvements are needed, and what additional services are desired. This approach, together with others intended to explore the perspectives of veterans who are not portal users, will help ensure that VA’s health information technology services are developed and enhanced to optimize the benefits to all VA patients. With impending changes to VA’s EHR platform, capturing veteran’s voices is more crucial than ever. More broadly, developing patient portals as an effective patient engagement strategy will require that UCD principles are employed to foster adoption and sustained use. In an era of finite resources, leveraging the “voice-of-the-customer” techniques helps ensure that the portal continues to meet patients’ needs in ways that enhance full participation in their own health care.