Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 23.10.17 in Vol 19, No 10 (2017): October

This paper is in the following e-collection/theme issue:

    Original Paper

    Veterans’ Preferences for Exchanging Information Using Veterans Affairs Health Information Technologies: Focus Group Results and Modeling Simulations

    1HSR&D Center of Innovation on Disability and Rehabilitation Research, James A. Haley VA Medical Center, Tampa, FL, United States

    2Department of Community & Family Health, College of Public Health, University of South Florida, Tampa, FL, United States

    3Veterans and Consumers Health Informatics Office, Veterans Health Administration, Department of Veterans Affairs, Washington, DC, United States

    4Department of Anthropology, University of South Florida, Tampa, FL, United States

    5Office of Health Informatics, Veterans Health Administration, Department of Veterans Affairs, Washington, DC, United States

    Corresponding Author:

    Jolie N Haun, EdS, PhD

    HSR&D Center of Innovation on Disability and Rehabilitation Research

    James A. Haley VA Medical Center

    8900 Grand Oak Circle

    Tampa, FL, 33637

    United States

    Phone: 1 813 558 7622



    Background: The Department of Veterans Affairs (VA) has multiple health information technology (HIT) resources for veterans to support their health care management. These include a patient portal, VetLink Kiosks, mobile apps, and telehealth services. The veteran patient population has a variety of needs and preferences that can inform current VA HIT redesign efforts to meet consumer needs.

    Objective: This study aimed to describe veterans’ experiences using the current VA HIT and identify their vision for the future of an integrated VA HIT system.

    Methods: Two rounds of focus group interviews were conducted with a single cohort of 47 veterans and one female caregiver recruited from Bedford, Massachusetts, and Tampa, Florida. Focus group interviews included simulation modeling activities and a self-administered survey. This study also used an expert panel group to provide data and input throughout the study process. High-fidelity, interactive simulations were created and used to facilitate collection of qualitative data. The simulations were developed based on system requirements, data collected through operational efforts, and participants' reported preferences for using VA HIT. Pairwise comparison activities of HIT resources were conducted with both focus groups and the expert panel. Rapid iterative content analysis was used to analyze qualitative data. Descriptive statistics summarized quantitative data.

    Results: Data themes included (1) current use of VA HIT, (2) non-VA HIT use, and (3) preferences for future use of VA HIT. Data indicated that, although the Secure Messaging feature was often preferred, a full range of HIT options are needed. These data were then used to develop veteran-driven simulations that illustrate user needs and expectations when using a HIT system and services to access VA health care services.

    Conclusions: Patient participant redesign processes present critical opportunities for creating a human-centered design. Veterans value virtual health care options and prefer standardized, integrated, and synchronized user-friendly interface designs.

    J Med Internet Res 2017;19(10):e359




    Patients often have busy schedules and competing priorities and want to control how and when they receive health services to meet their personal needs [1]. They often prefer to complete health-related tasks quickly and efficiently. In recognizing the needs of patients and their demand for convenient, continuous care, the Department of Veterans Affairs (VA) provides health information technology (HIT) resources that put veterans at the helm of their care. VA HIT complements traditional means of service delivery (eg, face-to-face, telephone, mail) and gives veterans the power to maximize the efficiency and convenience of their health care experience [2].

    VA’s remotely accessible HIT systems and apps include the My HealtheVet (MHV) patient portal, VetLink Kiosks, mobile apps, and telehealth services [3]. VA HIT supports veterans and their informal caregivers as active and informed proactive partners in their health care [4]. These tools help users manage appointments, keep track of medications, log personal health journals, record personal health care information and health measurements (eg, diet, physical activity, vital signs), communicate with their health care team, and access their electronic health record (EHR) [5].

    The VA has embraced the era of virtual health care delivery and initiated national efforts to redesign and reorganize HIT services. To ensure that HIT reflects veterans’ needs and supports their sustained use [5,6], the VA has leveraged human-centered design strategies [7]. The aim of this study was to provide a deeper understanding of veterans’ preferences for using HIT for managing chronic health conditions [8] and to inform VA HIT system design efforts. For the purposes of this paper, we focus on results from methods (ie, focus groups and pairwise comparison activities) that contributed to redesigning VA’s HIT systems and apps.


    This participatory study used mixed methods and included an expert panel and veteran participant focus groups. The study protocol has been previously published [8]. Expert panel members (EPMs) and veteran focus group participants provided descriptive information about VA and non-VA electronic health resources that veteran participants use for health care management. Pairwise comparison activities of HIT resources were conducted with both groups. Rapid iterative content analysis was used to analyze qualitative data. Descriptive statistics summarized quantitative data.

    Sample and Sampling

    Expert Panel

    Snowball sampling was used to identify VA providers, key operational representatives, and VA subject matter experts who could serve as EPMs. This study focused on VA HIT, so non-VA technologists were not included in the expert panel. Initial invitations were emailed to operational partners who were asked to represent their departments or technology-focused workgroups and to nominate other experts as needed to address gaps. EPMs participated in monthly meetings for 6 months. Their input, along with veteran participant data, led to the development of the VA HIT Systems Matrix. This novel tool describes the existing VA HIT system and identifies veteran participants’ vision for the future of an integrated VA HIT system. The VA HIT Systems Matrix was ultimately used to conduct a pairwise comparison activity [6].

    Veteran Participant Sample

    Purposive sampling yielded a sample pool for veteran participant recruitment efforts from two sites. We used administrative data to identify veterans who were registered for MHV, had completed the in-person process of authenticating their identity, and had opted to use Secure Messaging: 16,399 veterans in Tampa, Florida, and 1205 veterans in Bedford, Massachusetts. A greater number of veterans had registered for MHV and telehealth in Tampa than in Bedford, accounting for the difference in number of potential participants from each site. Next, we reviewed the list of potential veteran participants and identified 260 Tampa and 198 Bedford veterans who also used VA telehealth services. This ensured study participants had access to at least two forms of VA HIT resources.

    All 458 potential veteran participants were contacted and screened using a structured questionnaire. The structured screening questionnaire included items to determine whether potential participants met study criteria, including age (≥35 years of age), the presence of at least two chronic comorbid conditions (eg, diabetes, high blood pressure, chronic obstructive pulmonary disease) and use of specific VA HIT resources (including MHV, kiosks, mobile apps, telehealth). This tool also helped researchers determine if potential participants were high- or low-volume users of VA HIT. High-volume VA HIT users were defined as those using two or more types of VA HIT at least once a month. Low-volume VA HIT users were defined as those using fewer than two VA HIT platforms less than once a month, and using two or more other electronic resources at least once a month.

    We recruited approximately 10% of the sample pool. Ultimately, 47 veteran participants (44 male veterans and 3 female veterans) and one caregiver were grouped based on chronic health conditions and frequency of technology use (high, low). One female caregiver participated in a high-volume focus group. One female group (n=3) was convened to address woman’s health issues in addition to health conditions. This single group of females represented high-volume HIT users. Two other types of groups were formed: chronic conditions groups (n=7 groups) (eg, chronic obstructive pulmonary disease, diabetes mellitus, high blood pressure) and mental health groups (n=6 groups). These condition groups were then divided into high- and low-volume HIT use groups. See Table 1 for further details.

    Table 1. Focus group composition.
    View this table

    Data Collection and Asset Development

    Data were collected in two phases. In Phase 1, 48 focus group participants described their current use of VA and non-VA HIT and modeled their preferences for using these technologies in the future. A 16-item focus group guide incorporated free-listing (listing items based on their knowledge) and simulation modeling activities [8]. During the first set of focus groups, veteran participants discussed VA HIT system access, design, and functionality preferences in relation to their specific health care management tasks (eg, refilling prescriptions) and identified their vision for the future of an integrated VA HIT system. These data informed development of the aforementioned VA HIT Systems Matrix. This Matrix is a large detailed inventory of virtual platforms, their features, and contexts for use. It has been previously published and is omitted from this publication [9]. The Matrix provides information on the patient-facing platforms that are available to veterans (eg, MHV, mobile health, kiosks, telehealth), key system features (eg, Secure Messaging, Blue Button), access/availability, user groups, and context of use. The Matrix was used as an informational tool that helped veteran participants and EPMs complete the analytical hierarchy pairwise comparison process activity in Phase 2 of the study, further described below.

    Focus group data from Phase 1 also informed development of user personas, user scenarios, and low-fidelity representations (schemas, drawings, and process models) of participants’ system design and functionality preferences. User personas were “characters” developed to represent a veteran user in the scenarios. Process models provided a mapping strategy for developing interactive modeling simulations. A process model example is illustrated in Figure 1. The VA Human Factors Engineering (HFE) team used these assets, veteran comments provided through the MHV site, changes requested by VA clinicians, other veteran feedback provided by the VA Office of Connected Health, and an independent HFE study to create high fidelity, interactive, visual simulation models using iRise software [10]. HFE created the simulation with Structured Query Language (SQL) databases that enabled functionality similar to a live website. These functions included form submission, registration and credentialed sign-in with user recognition, live-updated data (dates or data previously submitted through forms), and validation error prompting. The simulation allowed a user to realistically use the prototype to support several representative veteran workflows such as refilling a prescription or canceling an appointment.

    The interactive simulations of redesigned VA HIT functioned on a variety of platforms (eg, mobile phone or tablet, desktop, kiosk) in test scenarios. These simulated models included mock app screens and webpages for platforms of interest (ie, Web, mobile, telehealth, kiosks). Participants provided feedback to refine modeled content in Phase 2 focus groups. This dynamic process of creating simulated models from participant data is illustrated in Figure 2.

    In Phase 2, participants from Phase 1 focus groups were divided into six Phase 2 focus groups based on participant availability. They reviewed the simulations of VA HIT and provided feedback on (1) accuracy of visual simulation models in capturing focus group input, (2) relevance of test scenarios, and (3) simulations’ design and functionality. Focus group facilitators gave a semi-scripted presentation that integrated veteran participant personas and user scenarios, multiple health management scenarios, and simulated prototypes of VA HIT on a variety of patient-facing platforms. Respondents’ reactions and experiences as they interacted with the simulations were audio recorded. Veteran participants also completed a similar pairwise comparison activity together at the end of each focus group. The data collection flow chart is illustrated in Figure 3. Participants then completed an analytical hierarchy pairwise comparison process activity [7]. This activity was conducted using a structured hierarchy of options for completing specific health care management tasks with VA HIT. The goal of the activity was for participants to select the best tool for completing a given task, by ranking alternatives. See the pairwise comparison worksheet in Figure 4.

    Figure 1. Process model example for tracking vitals.
    View this figure
    Figure 2. Process of creating simulated models.
    View this figure
    Figure 3. Data collection flow chart.
    View this figure
    Figure 4. Sample page from pairwise comparison worksheet.
    View this figure

    Data Management and Analysis

    Focus group data were transcribed and managed using the qualitative data analysis software program ATLAS.ti version 7.1 (ATLAS.ti Scientific Software Development). Data were analyzed in two stages [11]. The first round of coding included summarizing and data reduction from notes and transcripts into preliminary metadomains. Methods included deductive, structural coding with codes derived from the interview guide, and inductive, descriptive coding with codes that emerged from the data. A second round of coding allowed researchers to reduce coded data into meaningful domains and themes. The research team established an interrater reliability rate of 80%.


    Focus group participants represented a diverse veteran cohort and one caregiver who represented a veteran as a delegate. Participants were primarily male veterans with some college education, living with an average of six comorbid health conditions. Demographic data are presented in Table 2 and participant health conditions data in Table 3.

    Table 2. Participant demographics (N=48).
    View this table
    Table 3. Participants’ self-reported health conditions (N=48).
    View this table

    Focus Groups

    Current Use of VA Health Information Technologies

    All participants reported that electronic health tools and portals such as MHV and its component features are useful for managing health. Both types of user groups reported using (1) Secure Messaging (SM), a secure communication tool (like email) with VA health care providers, (2) Prescription Refills (Rx Refill), a secure online prescription refill program, and (3) MHV Appointments, an online resource that allows users to view past and future VA appointments as a list or on a customizable “Health Calendar” and to set up email reminders for upcoming appointments. High-volume HIT users were more likely to use telehealth, VA Mobile Apps, and additional MHV features, including the Blue Button, which allows veterans to view and download a copy of data from their EHRs, and the Veterans Health Library, an online veteran-focused library of reviewed health education resources. Sample quotes of current use of VA and non-VA HIT are included in Table 4.

    Table 4. Sample quotes of current use of VA and non-VA HIT.
    View this table
    My HealtheVet

    The most commonly used resource by participants was MHV, particularly the SM and Rx Refill tools. Participants valued SM to communicate with their health care teams. Frequent communication included lab management, appointments, medications, general health concerns, and specialty care requests. Many participants preferred SM because it maintains a record of their communication. High- and low-volume HIT user groups agreed SM was easy to use because it mimicked familiar email formats. They appreciated SM’s convenience, stating that providers responded quickly, and veterans could better manage their care while avoiding telephone waits or travel to their local VA facility. Barriers to using SM included providers who were not active SM users, perception of usability issues (ie, too many steps required to log in), and being less convenient than using personal email.

    Most participants liked the convenience of ordering prescription medications through Rx Refill. They requested refills in advance and could print Rx Refill pages for their personal records and community (non-VA) providers. High- and low-volume HIT users felt Rx Refill was complicated and did not adequately support management of several prescriptions. For example, users wanted notification when a prescription was going to expire, rather than scrolling through multiple online pages searching for refills or renewals. Many participants felt their Rx Refill page was cluttered with out-of-date prescriptions that impaired their ability to easily review current medications.

    Most participants used MHV Appointments, including reminders and a health calendar to check for future appointments and look up appointment instructions. They indicated this tool was not always current, and appointment notifications were often updated late or not at all. They suggested adding details about appointment location (eg, unit, room, floor) and accessible details about past appointments. Veteran participants were concerned that information they provided through the MHV Appointments platform was rarely relayed in a timely fashion to their care teams.

    Participants, particularly high-volume HIT users, reported using the Blue Button feature to print labs for community providers and personal records. Some participants reported difficulty using Blue Button, especially when accessing and interpreting lab results, and many felt that there were often too many pages to print out. Last, the Veterans Health Library was used by only one veteran participant in this study. Most veteran participants preferred easy to use non-VA sites for medical information (eg, WebMD).

    In general participants reported a desire for clean dashboard designs that were user friendly and easy to navigate. Modeling simulations that were prepared based on veteran participant‒reported preferences for the MHV home page and dashboard are illustrated in Figure 5.

    Figure 5. MHV home page and dashboard simulations.
    View this figure
    VetLink Kiosks

    At the time of the study, VetLink Kiosks were not widely available at the study sites. A kiosk is a veteran-facing touch screen device, found in VA clinics, that allows veterans to perform basic tasks such as checking into an appointment. Some participants used kiosks at appointment check-in, but believed kiosks had additional potential. Participants envisioned using kiosks to view their entire integrated EHR, search their medical records, and print information. They wanted the ability to print facility maps. They conceded privacy risks associated with accessing this information in view of the waiting room and suggested building a cubicle around kiosks to provide privacy.

    Veteran participants reported a desire for kiosks to be standardized, synchronized, and integrated with other VA HIT, particularly MHV. They reiterated their desire for a clean, user-friendly design. Modeling simulations that were prepared based on veteran participant‒reported preferences for the VetLink Kiosks are illustrated in Figure 6.

    Figure 6. VetLink Kiosk simulations.
    View this figure

    Veteran participants reported that telehealth services improved access to care. Veteran participants who used telehealth frequently used home telehealth to send vital signs to providers; however, they did not have access to previous submissions, making the tool ineffective for personal health monitoring. Participants reported preferring older telehealth equipment to the newer models of the phone telehealth system because the phone was too time consuming, though they did not provide specific details. A minority of veteran participants used video telehealth to communicate with providers for speech pathology and therapy appointments.

    The primary theme that emerged was veteran participants’ preference for synchronization, integration, and access to their data, particularly through MHV and Blue Button. Modeling simulations prepared using veteran participant‒reported preferences for access to their vital sign data are illustrated in Figure 7.

    Figure 7. Simulations of “Medical Record” containing Blue Button and Vitals/Readings features within My HealtheVet.
    View this figure
    Mobile Apps

    Few veteran participants used VA Mobile Apps, often due to reports of limited awareness of the available apps. It is also important to note a limited number of apps were available during this study, though many were in development and of interest to this study effort. There was a desire for convenient and easy-to-use apps. For those reporting use of the apps, Post-Traumatic Stress Disorder (PTSD) Coach, was most often cited, albeit infrequently. Veteran participants reported wanting mobile SM, appointment reminders, and Rx Refill apps or a single MHV app that integrated and synchronized all these features. Mobile apps preferences stemmed from a desire to have all health care management platforms conveniently located in a single place. Many veteran participants, particularly those with mental health issues, stressed the importance of creating secure mobile technologies.

    Telephone and Mobile Phone

    Participants used the telephone and mobile phones to supplement online activities. They communicate with providers, request prescription refills, and manage appointments. Phone use depended on status, urgency, and the level of accountability they wanted for a given issue. Participants reported a strong preference for using mobile phone technology to access MHV, mobile apps, and text alerts. Participants felt that although text messaging is not secure, there are appropriate uses for this technology such as appointment reminders and medication notifications. Modeling simulations for mobile phone designs and text features based on veteran participant preferences are illustrated in Figure 8.

    Figure 8. Modeling simulations of mobile phone designs and text features.
    View this figure

    Current Use of Non-VA Health Information Technologies

    High- and low-volume HIT users used Web browsers and search engines. Both groups preferred “clean,” “intuitive,” “simple to use” search engines that provided quick results. Participants with chronic health conditions used non-VA health technologies (eg, wearable heart, sleep monitors, pedometers) to better manage their condition. They used non-VA mobile apps to accomplish personal tasks such as tracking health parameters (eg, vital signs, weight, sleep patterns). These programs were described as “purposeful” and “tailored” to specific conditions and needs. High- and low-volume HIT users used multiple devices (eg, desktop, tablet, phone) noting the importance of quickly connecting to, and synchronizing information, across devices.

    Preferences for Future Use of VA Health Information Technologies

    Electronic Communications

    Veteran participants preferred exchanging information with providers electronically. Participants placed value on the use of SM to generate a record of communication that is accessible and accountable. They conceded that physician response time and adoption of this communication tool varied, and that the VA needs to implement mechanisms to overcome these barriers and improve SM effectiveness. Text messaging was thought to be the next logical platform for communicating with care teams. These tools would maintain the immediacy of a phone call and provide accountability by establishing a record of interaction. Sample quotes for future use of VA HIT are included in Table 5.

    Table 5. Sample quotes of preferences for future use of VA HITs.
    View this table

    System Design Preferences

    Participants drew inspiration from their personal use of technology and each other to model system design, functionality, and features. Models were created using large notepads, paper, and markers to demonstrate their preferences for system integration and synchronization. These data findings are described in the following sections.

    Access, Presentation, and Navigation

    A salient theme from the focus groups was participants’ desire for notifications and alerts. They reported preferences for notifications when secure messages are sent/received; appointments are made/changed; prescriptions are refilled, adjusted, added, or expired; labs are ordered or results are available; and progress notes are available. Participants felt strongly that incorporating text message, phone-based, or SM notifications into the appointment reminders platform was important to facilitate patient appointment adherence. For example, text messaging was the most desirable platform for receiving notifications. Participants also felt notifications via SM, phone, mail, and personal email accounts would be acceptable and beneficial. Figure 9 illustrates a modeled simulation of an MHV appointment manager based on participant-reported preferences.

    Medication management was also a top priority for participants. They had clear expectations for a user-friendly system that allowed management of many medications. Participants voiced a strong preference for medication lists that could be easily collapsed and expanded for managing information quickly and efficiently. Modeling simulations of this collapsing and expanding medication management system are illustrated in Figure 10.

    Last, participants wanted notifications and increased ease of access to lab results. Participants also reported problems interpreting lab results, voicing a strong preference for results to be illustrated in a user-friendly format with graphs and imagery. An example of a simulation based on their preference is presented in Figure 11.

    System Integration and Synchronization

    Participants reported a strong preference for all of their health information to be synchronized, integrated into their EHR, and accessible to them online. They desired changes to their electronic medical record to update within hours and be rapidly accessible. For example, a participant drew an image indicating a need for all HIT to be integrated, to exchange information provided by and to the patient across systems. Participant renderings were re-created to illustrate their preferences for system functionality (see Figure 12).

    Figure 9. Modeled simulation of My HealtheVet appointment manager.
    View this figure
    Figure 10. My HealtheVet Medication Manager in collapsed and expanded view.
    View this figure

    Some participants felt kiosks should also provide access to MHV records. In general, they felt synchronization and integration would significantly improve their health care management experience, particularly when managing appointments, medications, and vital signs. Figure 13 provides a conceptualization of their reported preference for providing vital sign information to their VA provider while also being able to immediately access and store that data on their personal software programs for self-care management.


    Veteran participants felt that standardizing the look, feel, layout, and navigation of all VA tools and platforms would make learning to use different technologies easier for diverse audiences. Participants also voiced preferences for universally recognized imagery, such as icons (eg, prescription, emergency, secure messaging), to be used to standardize the look of imagery across platforms.


    Veteran participants preferred a dashboard design for all VA HIT interfaces. The dashboard would be uncluttered, easy to use, and contain universally recognizable icons with large text. Many veteran participants said the look and feel should be based on commonly used software apps. One participant declared the dashboard should look like a car’s dashboard with “everything in one place.” When discussing MHV, participants also preferred to navigate from the homepage to features in one or two mouse clicks. They preferred that important information be centrally located, while news, updates, and other information be located at the bottom of the dashboard or omitted entirely. Standardization and design features captured across HIT platforms were simulated based on participants’ voiced preferences. Figure 14 presents how standardization of esthetics and design features across HIT platforms were simulated based on the group modeling activity.

    Figure 11. My HealtheVet Labs & Tests feature.
    View this figure
    Figure 12. Veterans’ drawing of system design preferences.
    View this figure
    Figure 13. Participant conceptualization of synchronized vitals between VA and personal software programs using telehealth technology.
    View this figure
    Figure 14. Visual model simulations of VA HIT standardization across various platforms (webpage, VetLink Kiosk, and mobile) based on data from group modeling activity.
    View this figure

    Participants preferred the secure nature of the initial in-person authentication currently required to allow veterans to access MHV advanced services such as SM and Blue Button. Those veteran participants that preferred online authentication wanted to provide their social security number and answer security questions to allow a remote but secure process for authenticating access.

    Delegation and Sharing Information With Community Providers

    Participants wanted to securely and electronically share their health care information with community providers and informal caregivers such as their spouses/partners, siblings, parents, or adult children. They appreciated the convenience of delegation. Many veteran participants reported that they already allowed family members to use their secure credentials to access their MHV accounts to help manage their health. Participants felt that they should be able to personalize levels of access to their EHRs and control who has access to different types of health information.

    Single Sign-On/Federated Credentialing

    Participants’ preferences regarding federated credentialing “single sign-on” varied depending on their knowledge of and proficiency in using technology. Commonly, those veteran participants who supported single sign-on explained that they had “password fatigue,” with general difficulty remembering usernames and passwords. While nearly all veteran participants acknowledged the expediency of a single-sign on or third-party credentialing mechanism, high-volume users were less likely to prefer this process, citing concerns about security. Low-volume users were less likely to understand federated credentialing but more likely to prefer it after it had been explained. Participants with security concerns were especially wary of credentialing via social media or private email accounts; however, they generally supported the idea of federated credentialing between government websites due to perceptions of high government Internet security standards.

    Accessing Information and Education About VA HIT

    When asked how the VA could improve awareness and use of VA HIT, most participants believed educating veterans about the availability and use of VA technologies is critical. Suggested education methods included peer-to-peer mentoring programs, instructional text in the form of a website or user guide, and instructional videos.

    Pairwise Comparisons

    Veteran participants preferred to access MHV resources such as SM, Appointment Reminders, Blue Button, etc, through a mobile app. However, there was uncertainty about the security, accessibility, and usefulness of mobile apps when managing health care. A slim majority of focus group participants preferred to access a VA electronic health resource using an Internet browser. Contrary to veteran participant preferences, EPMs overwhelmingly believed that veterans would prefer to use a mobile app to complete health care management tasks. In general, it was apparent that EPMs had a precise knowledge of which tool was designed for a given task. Differences between EPMs and veteran participant responses appeared to be largely based on EPM expert knowledge of resources, logistics, cost, and workflow issues.

    Veteran participants and EPMs agreed that SM is the preferred resource for completing a wide variety of health care management tasks. Veteran participants included SM in their lists of useful comparison resources more often than any other resource, and EPMs frequently ranked SM higher than most other available resources, suggesting that both groups considered it a useful tool for completing a range of health care management tasks even when it was not the most preferred option. Findings indicate that no single HIT solution is acceptable for the full range of health-related tasks, and a full range of options is needed. Preferences can change based on the individual or situation. Preferences among veteran participants and EPMs, and the resources they agreed on are presented in Table 6.

    Table 6. Pairwise comparison agreement between participant groups.
    View this table


    Principal Results

    The goal of this study was to inform the VA’s vision of an integrated HIT system from the veteran perspective [8,9]. Veteran participants value virtual health care delivery and are invested in having access to care anytime, anywhere [12]. Many of the current systems were designed to address a perceived need or fulfill a policy directive. Thus, the greatest value of this work is the development of veteran-driven high-fidelity modeling simulations and assets that illustrate user needs and expectations when using a HIT system and services to access VA health care services. These are critical contributions to the VA, a “patient-centered” organization that seeks to incorporate “the veteran voice” into all service areas, including HIT.

    Focus group findings illustrate the role of VA HIT in self-management of health care and chronic illness. It is logical that veterans with multiple chronic illnesses would rely heavily on technologies that support regular communication with providers (Secure Messaging), facilitate appointments (Appointment Reminders), and help with prescription management (RX Refill). For example, we now understand that tools like Rx Refill are vital to veterans with multiple chronic illnesses because they often manage many medications. Similarly, these findings provide important insights about barriers to use, along with suggestions for improvement. Veteran participants highlighted some of the functional improvements that could be made to help them manage a large volume of prescriptions, such as providing prescription expiration and refill notifications to help them maintain medication compliance.

    One major finding of this study reaches beyond the needs of veterans with multiple chronic illnesses. Our participants expressed a strong preference for standardized, integrated, and synchronized user-friendly interface designs. Although standardization has long been an issue of importance to usability and design efforts [13], improvements in standardization across VA HIT resources are needed to optimize effective usage. The participants in this study recognized VA HIT’s lack of visual standardization across platforms as a departure from many of the health management technologies available in the private sector and emphasized that improving the “look” of VA HIT was a critical step toward system modernization and promoting use. In addition, current system navigation and usability issues and concerns about security, back-up systems, and delegation can be successfully enhanced with a human-centered design approach. For example, veteran participants’ preferences suggest that navigation and security issues and issues of standardization may influence the potential for adoption and sustained use. When VA HITs appear and function consistently across platforms, it creates a level of recognition that promotes comfort. This could also impact the uptake of newly released VA HIT. It is likely that if new HITs have the same look, feel, and general functionality as other HIT, veterans will have fewer problems learning to use new systems and apps. There was evidence to suggest education on features such as federated credentialing, single sign-on, and associated security issues is needed to promote the acceptance of these features. To ensure veterans are aware of and know how to use VA HIT, the veterans in our study suggested just-in-time marketing and education about how to access and use VA HIT resources.

    The pairwise comparison focus group activity provided a unique way of discovering user preferences for use of VA technology platforms. Veteran participants expressed specific preferences for the platforms they wanted to use to accomplish specific tasks, its sense of urgency, and other situational contexts. Veteran participants and expert panel members agreed that a full range of options is needed, noting preferences can change based on the individual, the specific task, or the situation. In general, both veteran participants and expert panel members considered SM the most preferred resource. A slim majority of veteran participants preferred to access electronic health resources such as SM, Appointment Reminders, or Blue Button, using an Internet browser rather than a mobile app, in contrast to the belief of expert panel members who overwhelmingly believed that patients would prefer to use a mobile app. Veteran participants who did want to access resources through mobile apps expressed uncertainty about security, accessibility, and usefulness. Differences between veteran participant and expert panel member perspectives may be the result of panel member’s knowledge of logistics, cost, and workflow issues, as well as insights about future technology (mobile apps).

    Future research should inform VA’s vision for an integrated HIT system to include front-end veteran user experiences and outcomes. Specifically, research should evaluate best practices for supporting patients’ proactive and integrated use of VA HIT systems. In addition to assessing front-end veteran user experiences, veteran data also indicate that organization level research is needed to identify large-scale infrastructural consequences relevant to the supply and demand of the growing VA patient population. This research should assess the dynamic interaction of patient-provider electronic communication, and provider and team experiences, including workload and workflow, in order to ensure that the back-end systems and processes supporting the front-end veteran experience are operating effectively. Finally, system preferences such as single sign-on and delegation merit further investigation to better understand the feasibility, acceptability, and usefulness of these features within the current and evolving VA HIT system across traditional (eg, personal computers) and emerging (eg, mobile) technologies. Delegation has become increasingly important as the VA places more emphasis on engaging with community care providers and family care givers. Provision of comprehensive and consistent veteran health care rests on the veteran’s ability to securely and easily delegate access to medical records and virtual health services.

    Comparison With Prior Work

    This work builds on previous work exploring user experiences on individual HIT platforms and tools within and beyond the VA. However, to our knowledge this is the first study to look at user experience across an enterprise-wide system of VA HIT platforms and tools. The unique contribution of this work is its comprehensive approach to looking at currently available VA HIT capacity and emerging functionality. As such, the modeling simulations produced in this work are veteran driven and can inform ongoing VA HIT redesign initiatives.


    Although this study underscored veteran preferences for using HIT and offered their recommendations for system improvements, it had some limitations. First, the study reports findings from two specific VA facilities. While participants were a representative purposively sampled group [14], additional insights may be gained by expanding this assessment to other VA facilities and veteran populations. Second, findings are primarily relevant to VA HIT systems and technologies but may be useful for the development and redesign of other tethered HIT systems. Third, current technological infrastructure capacity was not a primary focus and thus some desired changes may not yet be technically possible. Fourth, we purposively recruited participants who were invested users of two or more platforms; we may have missed valuable data that may have represented non-invested users. Finally, we included veterans with chronic conditions because they are more likely to leverage electronic resources to manage their health care, as such, we may have missed valuable data that may represent healthier participants [15,16].


    This is one of the few published studies to aid in the development of an integrated system of patient-facing HIT resources within a large health care system. The findings from this study have already had a direct impact on the incremental redesign of the My HealtheVet patient portal and the prioritization of approaches that provide integration between VA HIT platforms. Future research can inform the ongoing development of VA’s integrated HIT system, to include front-end patient user experiences and back-end workload and workflow. Future work should evaluate best practices for supporting consumers’ proactive and integrated use of VA HIT systems. Though this research lends itself to recommendations for future research, our aim in completing this work was to inform a user-centric perspective to assist ongoing development, redesign, and research efforts. These assets were developed from a veteran-centric perspective to support the use of VA’s dedicated resources to materialize the findings in ongoing VA HIT redesign efforts. Organizations beyond the VA can benefit from using a similar approach and may discover the findings useful in designing human-centered HIT systems.


    The Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Health Services Research and Development Service, and National eHealth Quality Enhancement Research Initiative Coordinating Center (RRP 12-495) supported the development of this manuscript. This manuscript was also supported in part by the Center of Innovation for Disability and Rehabilitation Research at the James A. Haley Veterans Hospital.

    The contents of this manuscript do not represent the views of the Department of Veterans Affairs or the US Government.

    The authors would like to acknowledge the efforts of Robert M. “Goose” Gosline, Communications Specialist and My HealtheVet Coordinator at James A. Haley Veterans Hospital for his significant contribution to the recruitment for and success of this project.

    Conflicts of Interest

    None declared.


    1. Nazi KM, Hogan TP, Wagner TH, McInnes DK, Smith BM, Haggstrom D, et al. Embracing a health services research perspective on personal health records: lessons learned from the VA My HealtheVet system. J Gen Intern Med 2010 Jan;25 Suppl 1:62-67 [FREE Full text] [CrossRef] [Medline]
    2. Nazi KM. Structures and processes in health care systems: Lessons learned from the VA HealtheVet portal. In: Duffy VG, editor. Advances in Human Aspects of Healthcare (Advances in Human Factors and Ergonomics Series). Boca Raton, FL: CRC Press; 2012:165-178.
    3. Veterans Health Administration. Connected Health   URL: [accessed 2017-07-12] [WebCite Cache]
    4. Woods SS, Schwartz E, Tuepker A, Press NA, Nazi KM, Turvey CL, et al. Patient experiences with full electronic access to health records and clinical notes through the My HealtheVet Personal Health Record Pilot: qualitative study. J Med Internet Res 2013 Mar 27;15(3):e65 [FREE Full text] [CrossRef] [Medline]
    5. Hogan TP, Wakefield B, Nazi KM, Houston TK, Weaver FM. Promoting access through complementary eHealth technologies: recommendations for VA's Home Telehealth and personal health record programs. J Gen Intern Med 2011 Nov;26 Suppl 2:628-635 [FREE Full text] [CrossRef] [Medline]
    6. Nazi KM. The personal health record paradox: health care professionals' perspectives and the information ecology of personal health record systems in organizational and clinical settings. J Med Internet Res 2013 Apr 04;15(4):e70 [FREE Full text] [CrossRef] [Medline]
    7. Harte R, Glynn L, Rodríguez-Molinero A, Baker PM, Scharf T, Quinlan LR, et al. A human-centered design methodology to enhance the usability, human factors, and user experience of connected health systems: A three-phase methodology. JMIR Hum Factors 2017 Mar 16;4(1):e8 [FREE Full text] [CrossRef] [Medline]
    8. Haun JN, Nazi KM, Chavez M, Lind JD, Antinori N, Gosline RM, et al. A participatory approach to designing and enhancing integrated health information technology systems for veterans: protocol. JMIR Res Protoc 2015 Feb 27;4(1):e28 [FREE Full text] [CrossRef] [Medline]
    9. Haun JN, Chavez M, Nazi KM, Antinori N. Developing a health information technology systems matrix: A qualitative participatory approach. J Med Internet Res 2016 Oct 06;18(10):e266 [FREE Full text] [CrossRef] [Medline]
    10. Kabel M, Cook A, Robbins J. HFE Comments and Findings: MHV High Impact Wins Focus Group. Virginia: Veterans Affairs; 2014.
    11. Miles MB, Huberman AM, Saldaña J. Qualitative data analysis: a methods sourcebook. Thousand Oaks, CA: SAGE Publications Ltd; 2014.
    12. Haun JN, Patel NR, Lind JD, Antinori N. Large-scale survey findings inform patients' experiences in using secure messaging to engage in patient-provider communication and self-care management: A quantitative assessment. J Med Internet Res 2015 Dec 21;17(12):e282 [FREE Full text] [CrossRef] [Medline]
    13. Nielsen J. Nielsen Norman Group. 1995 Jan 01. 10 Usability Heuristics for User Interface Design   URL: [accessed 2017-08-22] [WebCite Cache]
    14. Guest G, Bunce A, Johnson L. How Many Interviews Are Enough? Field Methods 2016 Jul 21;18(1):59-82. [CrossRef]
    15. Haun JN, Lind JD, Shimada SL, Simon SR. Evaluating Secure Messaging from the veteran perspective: Informing the adoption and sustained use of a patient-driven communication platform. Annals of Anthropological Practice 2014 Sep 02;37(2):57-74. [CrossRef]
    16. Haun JN, Lind JD, Shimada SL, Martin TL, Gosline RM, Antinori N, et al. Evaluating user experiences of the Secure Messaging tool on the Veterans Affairs' patient portal system. J Med Internet Res 2014 Mar 06;16(3):e75 [FREE Full text] [CrossRef] [Medline]


    EHR: electronic health record
    EPM: expert panel members
    HIT: health information technology
    MHV: My HealtheVet
    SM: Secure Messaging
    VA: Department of Veterans Affairs

    Edited by G Eysenbach; submitted 22.08.17; peer-reviewed by C Turvey; comments to author 15.09.17; revised version received 28.09.17; accepted 28.09.17; published 23.10.17

    ©Jolie N. Haun, Margeaux Chavez, Kim Nazi, Nicole Antinori, Christine Melillo, Bridget A Cotner, Wendy Hathaway, Ashley Cook, Nancy Wilck, Abigail Noonan. Originally published in the Journal of Medical Internet Research (, 23.10.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.