Published on in Vol 18, No 5 (2016): May

Applying Multiple Methods to Comprehensively Evaluate a Patient Portal’s Effectiveness to Convey Information to Patients

Applying Multiple Methods to Comprehensively Evaluate a Patient Portal’s Effectiveness to Convey Information to Patients

Applying Multiple Methods to Comprehensively Evaluate a Patient Portal’s Effectiveness to Convey Information to Patients

Original Paper

1Department of Health Behavior and Policy, Virginia Commonwealth University, Richmond, VA, United States

2Department of Family Medicine and Population Health, Virginia Commonwealth University, Richmond, VA, United States

3Department of Communication, George Mason University, Fairfax, VA, United States

Corresponding Author:

Jordan M Alpert, PhD

Department of Health Behavior and Policy

Virginia Commonwealth University

830 E Main St

Richmond, VA, 23219

United States

Phone: 1 8046284790

Fax:1 8046281233


Background: Patient portals have yet to achieve their full potential for enhancing health communication and improving health outcomes. Although the Patient Protection and Affordable Care Act in the United States mandates the utilization of patient portals, and usage continues to rise, their impact has not been as profound as anticipated.

Objective: The objective of our case study was to evaluate how well portals convey information to patients. To demonstrate how multiple methodologies could be used to evaluate and improve the design of patient-centered portals, we conducted an in-depth evaluation of an exemplar patient-centered portal designed to promote preventive care to consumers.

Methods: We used 31 critical incident patient interviews, 2 clinician focus groups, and a thematic content analysis to understand patients’ and clinicians’ perspectives, as well as theoretical understandings of the portal’s use.

Results: We gathered over 140 critical incidents, 71.8% (102/142) negative and 28.2% (40/142) positive. Positive incident categories were (1) instant medical information access, (2) clear health information, and (3) patient vigilance. Negative incident categories were (1) standardized content, (2) desire for direct communication, (3) website functionality, and (4) difficulty interpreting laboratory data. Thematic analysis of the portal’s immediacy resulted in high scores in the attributes enhances understanding (18/23, 78%), personalization (18/24, 75%), and motivates behavior (17/24, 71%), but low levels of interactivity (7/24, 29%) and engagement (2/24, 8%). Two overarching themes emerged to guide portal refinements: (1) communication can be improved with directness and interactivity and (2) perceived personalization must be greater to engage patients.

Conclusions: Results suggest that simple modifications, such as increased interactivity and personalized messages, can make portals customized, robust, easily accessible, and trusted information sources.

J Med Internet Res 2016;18(5):e112



There is significant progress to be made in creating digital communication platforms that improve health outcomes [1]. While patient portals have been lauded as a method to enhance health communication [2], and they have been found to be helpful, their impact has not been as profound as anticipated. Yet patients are still enthusiastic about the portals’ capabilities to assist in managing their health [3]. In one study, 43% of patients believed that apps, such as portals, could improve relationships with doctors, 48% would feel more in control of their health, and 40% would be would encouraged to ask physicians more questions [4]. Evidence has demonstrated that portals contribute to improved health outcomes by increasing cancer screenings, especially when information was explained in lay language, used personalized recommendations, and provided educational resources [5]. Still, clinicians report dissatisfaction [6] and patients are underwhelmed with the design and functionality of portals [7].

Incorporating immediacy into portal design has the ability to increase their usability and importance as health communication resources. Immediacy refers to features that promote physical and emotional closeness, caring relationships, authenticity, and enthusiasm [8]. The construct has a rich tradition focusing on face-to-face communication in education [9,10], psychological counseling [11,12], and health care delivery [13,14]. However, immediacy has only recently been applied to digital health communication. Incorporating immediacy into the design of health communication tools can better engage, involve, and motivate patients to promote their health and well-being [15]. However, thus far there has been little evidence to demonstrate that portals can create a sense of immediacy [16].

Our study explored how a unique set of methods—critical incident reports from patients, focus groups of clinicians, and thematic analysis—can be used to evaluate and better inform the design of patient portals. We analyzed one exemplar patient portal, MyPreventiveCare [17], as a test of concept. MyPreventiveCare, a highly functional, prevention-focused online resource, was designed specifically to activate and engage patients around preventive care [18]. It is used by 12 practices in Virginia in the United States and reaches over 82,000 patients, but is being extended to an additional 300 practices in 15 states through a series of grants. In addition to providing laboratory results and viewing information from the medical record, the innovative portal customizes content based on hundreds of demographic, historical, behavioral, and clinical variables to make personalized recommendations and provide educational material based on current guidelines using content from, a credible online health information resource from the National Health Information Center (Washington, DC). The portal provides information about needed cancer screenings, how to better monitor chronic conditions, and how to improve health behaviors. Nearly half (43.4%) of patients registered on MyPreventiveCare have logged in to the portal in the past year and frequently visit the following areas: laboratory results, medical record information, and preventive care recommendations. The average patient accesses the portal almost 4 times per year.

To sufficiently assess the test portal, we used multiple methods to collect and analyze data. Using multiple methodologies in health science research helps researchers view problems from multiple perspectives to enrich the meaning of a singular viewpoint [19] and contributes to developing a more complete understanding of a problem [20]. We describe the methodology for conducting interviews, focus groups, and the thematic analysis.



Age demographics for current users of the portal are as follows: 18–24 years (3980/72,362, 5.50%), 25–34 (13,387/72,362, 18.50%), 35–44 (16,281/72,362, 22.50%), 45–54 (16,643/72,362, 23.00%), 55–64 (12,302/72,362, 17.00%) and ≥65 (9769/72,362, 13.50%). More females (41,246/72,362, 57.00%) use the system than males (31,116/72,362, 43.00%) and in fact, the most frequent users are women 45–54 years old. Although women 45–54 years old is the largest demographic group of users, criteria for interviews included patients ranging from 18–79 years, to generate as many viewpoints as possible.

We used naturalistic intercept sampling techniques [21], such as purposeful sampling [22] and convenience sampling [21], to recruit active users, age 18-79 years, who accessed the system at least once in the past year. To enhance recruitment, efforts were made to make participation in the interviews convenient for patients who already were at the clinic waiting for their doctors’ appointments. Patients who met eligibility and had an upcoming appointment were contacted via telephone 1 week in advance to schedule an interview before the appointment with their physician. In addition, the medical staff identified eligible patients and referred them either before or after the appointment with their physician. We recruited participants in the fall of 2014 and winter of 2015 in 2 primary care practices that belong to the Virginia Ambulatory Care Outcomes Research Network. The study was described to patients in a private conference room, and then informed consent was reviewed. Scheduling interviews with patients around impending appointments over the course of several weeks allowed for a diverse cross-section of patients to be recruited.

Critical incident technique (CIT) [23] was the main method of inquiry, which is a qualitative research method with strong exploratory and investigative abilities [24-26]. The technique was designed as a flexible set of principles with the goal of identifying incidents that the users considered positive or negative and to probe participants about their experiences. The CIT is especially advantageous to the evaluation of a patient portal because it allows users to reflect on the most meaningful events from their experience using the system. Unlike usability studies, which solely focus on the way an individual operates a system, CIT allows both patients and clinicians to identify instances in which the portal affected their lives outside of directly using the system. CIT has been used to analyze quality of care [25] and is a common method in the application of health care services [27].


During interviews, participants were asked to recall their best and worst experiences using the portal. These 2 broad questions framed the discussion, allowing the interviewer to focus on the incidents mentioned and investigate their significance. Interviews ceased and saturation was achieved when keywords and phrases were frequently repeated, participants’ discourse was similar, and recurrent meanings were discovered [28]. All discussions were audio recorded and subsequently transcribed. The average interview time was 14 minutes in length.

Analytical Process

To ensure objectivity of the incidents collected, we developed a set of specific rules that specified whether an incident was positive or negative based on (1) the situation of the incident, (2) relevance to the general aims of the study, and (3) importance of the incident’s effect on the aims [23]. Data analysis was guided by the tradition of CIT and the constant comparative method [29], in which 2 researchers (JA, GK) independently read each transcript, exploring for prevalent themes. Initially, any incident that could be considered either positive or negative was collected. Upon further analysis, the researchers discussed examples of positive and negative incidents, thereby refining the qualifications to be considered an incident worthy of inclusion. Next, the researchers shared observed themes and formed categories by creating an aggregated codebook. If discrepancies occurred, the researchers referred to the rules and aims of the study and discussed their perspective until consensus was met. Each researcher once again independently analyzed the data and assigned a code that best captured the significance of the incident. The researchers met to review their findings frequently and discussed any discrepancies that arose until they reached a consensus around a particular theme. To assess the validity of the data, we initiated respondent validation, or member checks [30], with several participants to confirm the interpretations of the findings.

Focus Groups


We conducted 2 focus groups with clinicians in 1 Northern Virginia medical office. The administrator of the portal emailed providers working in the medical office and made an announcement at a monthly meeting about participation in the study. Participation was voluntary and scheduled during providers’ lunch hour. The first focus group included 8 physicians (2 men and 6 women), comprising 4 residents and 4 full-time physicians. The second focus group consisted of 5 participants (4 women and 1 man), 4 nurses and 1 emergency medical technician. The average length of both focus group sessions was 51 minutes.


We used the CIT to allow clinicians to concentrate on the extreme positive and negative functions of the portal. A printed list of general questions was presented at the beginning of the focus group session. Questions were grouped into the following domains: positive experiences, negative experiences, and how the portal positively or adversely affected the communication process with patients and staff. The questions were semistructured, to allow for organic, flexible conversations that stimulated discussion [22]. Each focus group session was audio recorded and transcribed. For transcripts involving the focus group data, we used the same analytical process previously described for interviews.

Thematic Analysis of the Portal

We conducted the third method, thematic analysis, after interviews and focus groups were completed. We used patients’ and clinicians’ perspectives as a lens to analyze the content found on the portal, as well as by using key factors of immediacy identified in previous research [15,31,32]: user engagement, personalization, interactivity, enhances understanding, and motivates behavior change (definitions can be found in Table 1 [33-40]). We conducted the thematic analysis by reviewing the portal, searching for specific patterns related to immediacy. To strengthen claims of the thematic analysis, we incorporated numerical results, which enabled the amount of evidence in the data to be quantified to support the conclusions [41].

Table 1. Immediacy definitions for thematic analysis of the patient portal MyPreventiveCare
Immediacy featureDefinition
InteractivityDefinitions of interactivity typically focus on 2 measures: (1) 2-way flow of information and (2) rapid exchange of information. Other definitions include control as the main component of interactivity [33], meaning that participants should be able to exercise control over the communication exchange. For the purposes of this study, interactivity includes all 3 components and is considered to be reciprocal and synchronous communication that offers active control [34].
PersonalizationAccording to e-commerce websites, personalization is “the adjustment and modification of all aspects of a website that are displayed to a user in order to match users’ needs and wants” [35]. Although the portal is not an e-commerce website, the previous definition was thoroughly explicated and is relevant to this study.
EngagementCombining definitions from the fields of marketing [36], media [37], and psychology [38], engagement is considered the level of an individual’s physical, cognitive, and emotional involvement or connection with a specific medium.
Motivates behaviorMotivators are triggers, prompts, cues, or calls to action that encourage a user to take action [39].
Enhances understandingFactors that enhance understanding, similar to relational understanding, in which an individual knows both what to do and why [40].

Analytical Process

The first author (JA) logged in to the portal as a test user and analyzed 27 separate content pages, of which 24 were evaluated. The home page, “Dashboard,” and “Library” pages were not analyzed because they only functioned as navigational pages. Every webpage was examined individually to determine the prevalence of each immediacy characteristic. A checklist with each immediacy feature’s definition was designed and individual webpages were meticulously scrutinized using the checklist. After an initial review, each page was checked a second time to confirm initial findings. After the first author conducted the analysis, each of the authors gave input in regard to the identification of themes. In particular, the portal’s administrator reviewed the findings and provided feedback on actual patient and provider experiences. The first author then reevaluated each individual webpage taking into account the administrator’s perspective, as well as the theoretical framework of immediacy features.

The use of multiple methods (interviews, focus groups, and thematic analysis) and recruitment of key stakeholders (patients and clinicians) provided triangulation by using methods with different strengths and limitations to support a single conclusion [42].

This study was given full institutional review board approval by the George Mason University Office of Research Integrity & Assurance. Participants signed an informed consent document and, to ensure confidentiality, audio files were only accessed by the researcher, and personal information, such as names, were de-identified upon transcription.

The study included 44 total participants in 31 patient interviews (18 women and 13 men) and 2 focus groups (13 clinicians). We collected a total of 142 incidents, 102 negative and 40 positive.

Patient Interviews

Incidents classified as positive (31/113, 27.4%) were outweighed by negative incidents (82/113, 72.6%). The following themes were most salient throughout positive and negative incidents cited by patients.

Positive Incidents

Patient interviews revealed 3 main categories of the portal’s usefulness: (1) the ability to instantly access medical information, (2) availability of clear health information, and (3) patient vigilance.

Instant Access

Nearly half of all positive incidents were associated with the instantaneous retrieval of medical information. For example, patient #1, a busy mother, anxiously awaited test results but was unable to call the office during business hours. She appreciated how she could log in to the portal in the evening and look up the results.

Similarly, most patients complained that obtaining laboratory results was a chore involving endless telephone calls, but the portal streamlined the once-laborious process. Patient #8 described how he used to get results before the portal was available. He said, “The office doesn’t have time to call you, or they call, and they don’t get you and leave a message, and you call back and leave a message.”

Clear Health Information

Not only did patients appreciate the ability to get laboratory results, but also the portal’s design made it easy to navigate. Upon logging in, patients were presented with a dashboard featuring large, colorful icons used as the website’s primary navigation. Referring to the portal’s interface, patient #30 commented, “It is very icon-driven and it’s all very intuitive, so I don’t find it complicated to use.” In addition, throughout every section of the website, a dictionary is available to look up challenging terminology.

Patient Vigilance

The ease of the portal’s navigation contributed to patients being able to carefully monitor their health. Patients used information found on the portal to gauge their health status. Patient #4 monitored her laboratory results to determine whether “medications were working or if there was a problem somewhere.” Correspondingly, patient #11 always checked laboratory results as soon as they were available and said, “If my labs are getting worse, or even if they’re normal, I can make adjustments and monitor it closer.” The availability of health information, such as laboratory results, allowed patients to check differences from previous tests, while prevention recommendations, like getting the flu shot, reinforced efficacious behavior and served as a reminder.

Negative Incidents

Negative incidents were classified into 4 main categories: (1) standardized content, (2) the desire for clinicians to communicate directly with the patient, (3) website functionality, and (4) difficulty interpreting laboratory data.

Standardized Content

Many patients were unaware that content was personalized based on their electronic medical record and self-reported data. For instance, patient #3 said, “it felt generic. I don’t know if it was specific to my medical history, but it didn’t feel personal.” Patient #12 agreed and said, “I’d want something more focused on me as opposed to something generic like this.”

Desire for Direct Communication

After reading the portal’s content, many patients craved information directly from the clinician. For instance, patient #15 wondered, “Can the doctor and the medical provider put what they would propose as next steps?” Although the content that appeared was generated from updates the clinician made to the patient’s record, the language used did not reflect that the content was personally delivered by a clinician. This sentiment was best expressed by a retired woman, patient #10, who was inspired with new questions and concerns after viewing content from the portal. She often consulted advice nurses, but since they were no longer available, she yearned for the portal to be a substitute.

In addition to the desire to interact directly with clinicians, many patients wanted a resource that would be beneficial between scheduled office visits. Patient #4 stated, “once in a while something goofy will happen, so it would be nice to come in here (portal) and read the things that I need.” Similarly, patient #2’s blood pressure often wildly fluctuates and during those times, she thought, “it would be nice if I could say, is this something I should worry about?”

Website Functionality

The portal was susceptible to problems experienced by many websites, such as issues logging in and server crashes. Indeed, 20% (16/82) of all negative patient incidents involved an instance in which the portal did not function correctly. Patient #12 remembered a time when he had difficulty recovering his password. He said, “I couldn’t remember what my password was. Even after I got the password, when I went to log in again, it wouldn’t let me.”

When patients were able to log in, data entry mistakes or database errors sometimes caused the portal to inaccurately report results. Patient #5 said, “It’s been hit or miss as far as having the information being correct.” Also, information was not updated to appropriately reflect their health status. Patient #28 cited, “I kept getting these flash things that said I was overdue on a pap exam. I had just been in in April, but it kept flashing at me like I was a bad girl.”

Laboratory Data Interpretation is Difficult

Patients appreciated the portal’s preventive care messages, such as a recommendation to get a mammography, but experienced difficulty interpreting diagnostic laboratory information. This accounted for 11% (9/82) of negative incidents. For instance, patient #15 referred to the “Watch Your Weight” page and said, “One question I have is this BMI [body mass index]. Is it just sort of a number pulled out of the air or is it in fact appropriately calculated?” He wanted to better understand what the number actually meant to “take the number seriously.” Similarly, some patients had trouble deciphering the results of the blood sugar section, as exemplified by patient #27, who examined the chart on the webpage and wondered, “It was measured on September 3rd. I have a value of 100 and my goal is to be less than 126, so I should be good? I don’t know what that symbol means. [Orange icon indicating a marginal score]. Why would I be marginal?”

Clinician Focus Groups

We combined data from the 2 focus groups, one with physicians and another with rooming staff. Clinician incidents were comparable with patient data, with 69% (20/29) negative and 31% (9/29) positive.

Positive Incidents

Positive incidents among clinicians were classified into 2 main areas: (1) patients feel empowered (5/9, 56%) and (2) the portal can generate office efficiency (4/9, 44%).

Develops Patient Empowerment

Over half of all positive incidents involved the belief that patient access to medical information was beneficial. A nurse recalled, “I work with patients who have jobs that take them out of the country and the fact that they have this information is awesome.” Before the introduction of the portal, nurses would review laboratory results over the phone, but patients did not have access to the data as a reference. Furthermore, clinicians observed higher levels of patient motivation after interacting with the portal. A nurse said, “[Patients] will use the portal to track things, like ‘my cholesterol wasn’t that good, so I need to increase my exercise’, and next time, they’ll come in and say, ‘I added another day to my exercise routine and I’m really anxious to see how my cholesterol is now.’”

Due to increased levels of motivation, patients experienced more productive office visits. A physician noticed that the portal “helps start a conversation when there’s so much to cover.” Another physician confirmed that notion and said the portal created “patient led agendas rather than the physician leading the agenda.” He continued, “I’ve had patients who have already looked at their laboratory results before they come in.” Physicians noticed that patients altered their behavior and even researched possible treatment options before their appointment.

Generates Workplace Efficiencies

Although many more negative incidents focused on how the portal has the potential to create additional work, positive incidents highlighted how the portal contributes to a more efficient working environment. For instance, reminders delivered to patients via the portal lessened the need for unit clerks to call and remind patients about upcoming examinations. A physician agreed that reminders were particularly useful because they prevented unit clerks from interpreting medical information. In addition, less time was devoted to playing “phone tag” because patients could correct errors, such as 1 incident in which a patient’s file stated that her last colonoscopy was 12 years ago, even though the procedure was performed 5 years ago.

Negative Incidents

Clinicians’ incidents overlapped with patients’, and in many cases, reinforced what patients experienced. Negative incidents fell into 3 main categories: (1) lack of feedback, (2) fears that the portal can increase workload, and (3) inappropriate use of the system.

Lack of Feedback

Clinicians considered the portal a valuable tool, but acknowledged that it contained flaws. Perhaps the biggest flaw, which accounted for 40% (8/20) of negative incidents, was that clinicians could not confirm whether patients viewed or understood information that was input into the portal. A nurse summarized the problem by describing how she typed the analysis of laboratory results or medication directions, but was unaware whether patients either saw the information input or understood its meaning.

In other scenarios, clinicians received inquiries from patients, but the portal was not deemed an appropriate forum to discuss the matter. For example, a resident recalled the following predicament: “You get this message and you want to answer it...but I find that if that question pushes the boundaries of what I should do outside of an office visit, maybe I should bring them in? But they’re asking for it, so I send the anti-biotic I otherwise wouldn’t.”

Another issue that compressed communication between patients and clinicians was the belief that the portal was not equipped to handle complex communication. A doctor brought up the issue of cancer screening and said, “The problem is there are different ways to do it. We all know we should screen for colon cancer. When I see a patient in person, I ask them do you want to do a colonoscopy or annual stool test?” According to clinicians, a much richer communication platform was needed to conduct meaningful conversations.

Fears of Increased Workload

Although clinicians cited specific instances in which the portal generated efficiency in the office, there was still speculation that increased use of the portal would create additional work. Nurses were worried that patients would call the office at greater frequencies because sometimes “patients look at laboratory results and one thing is a little off, so they freak out and they’re calling us and asking.” Physicians were also concerned about devoting more time to addressing patients’ questions because, as one doctor said, “It is frankly time we don’t get reimbursed for.” Since physicians did not receive confirmation that the patient viewed the message, they had to take the time to call the patient and ensure the information was communicated.

Inappropriate Use of the System

Surprisingly, clinicians acknowledged that they sometimes purposefully contributed complex medical jargon. For instance, nurses said that younger doctors entered sophisticated terms because they wanted to impress their supervisors who may see the information they input. Physicians with more seniority were also at fault. Nurses complained that many doctors wrote messages meant for nurses and were unaware that patients were also capable of viewing them.

Sometimes, physicians recognized that content on the portal was not written in a sensitive manner. A doctor said, “I’ve had a few patients who get the health maintenance reminders and say, ‘my portal says I’m fat.’” Other doctors felt that the content on the portal may be suitable for the average patient, but might be beyond comprehension for patients who are not native English speakers.

Thematic Analysis of the Portal

On the basis of attributes of immediacy (interactivity, personalization, engagement, motivates behavior, enhances understanding), we examined the portal as to whether the characteristics were present (summarized in Table 2).

Table 2. Presence (+) or absence (–) of immediacy characteristics in the patient portal MyPreventiveCare.
Enhances understandingPersonalizationMotivates behaviorInteractivityEngagement
Preventive care recommendations

Summary page++++

Take aspirin+++

Get tested for diabetes+++

You have high blood pressure+++

Get a tetanus shot+++
Other recommended behavior


Cervical cancer++++

Colon cancer testing++++






Pneumonia vaccine+++

Flu vaccine++

Bone density+++
Laboratory results

Your labs+
Edit information section

Update page+++++


Setting priorities

Prevention topics A–Z+++

Self-management tools+++

Help page++
Contact us

Contact pagen/aa

Characteristics present, n (%)19/23 (78%)18/24 (75%)17/24 (71%)7/24 (29%)2/24 (8%)

an/a: not applicable.

Enhances Understanding

This category measured whether content allowed patients to understand not only what to do, but why to do it. We did not review the “Contact” page because its content was not applicable. This section received the highest score, with 78% (18/23) of the portal’s content contributing to enhancing understanding.

The way in which information was presented was integral to enhancing patients’ understanding. Patient #28 commented how the bulleted lists made it easier to process recommendations. Moreover, the use of graphics supported the text it accompanied. For example, an image on the “Exercise” page of a man playing basketball contributed to reinforcing healthy behavior, while photographs of individuals eating fresh vegetables on the “Diet” page emphasized the concept of healthy eating and made it more appealing. Showcasing positive behaviors supplemented the information presented and served as a model to apply recommendations. Other factors that enhanced understanding were detailed explanations. For instance, on the “Take aspirin” page, an initial header outlined the patient’s health status, such as whether they smoked and whether they were taking an aspirin dosage. Next, a brief explanation on the benefits of aspirin appeared, followed by the criteria for whether aspirin is appropriate, next steps, and several related links to more information about aspirin, as well as heart disease. Similarly, T-scores were described in the “Bone density” section, including how weight influences blood pressure. Throughout the portal, simple and direct language was used, demonstrated by the average word length at only 6 letters, which contributes to enhancing patient understanding.


Personalization was determined by an enhanced sense of inclusion and cooperation to match users’ needs and wants. The main way personalization was achieved came through language, such as using pronouns (“we” or “you”), as well as providing unique information applicable to an individual’s life. For instance, recommendations were offered, like “You are due to get another tetanus shot now” and, after listing a patient’s medical information, the following text was generated: “These conditions place you at higher risk for heart disease.” The portal’s customized content contributed to 75% (18/24) of pages having the quality of personalization.

Although we gave credit for personalization, terms like “you” and “your” were inconsistent. On the “Your labs” page, information was written in a very clinical manner, negating past attempts at personalization. The text read, “Patient would like a mammogram,” instead of using “you” or the patient’s name. This was especially true on the dashboard, which was the first page that appeared when a patient logged in. At the top, it read, “You are here,” instead of immediately establishing personalization by using the patient’s name.

Other areas of the portal lacking personalization were the “Dictionary” and “Prevention topics” pages, which listed topics that could be irrelevant to patients, such as information about cervical cancer to male patients. Lastly, stock photographs of physicians were used throughout the portal instead of photos of clinicians familiar to patients.

Motivates Behavior

Aspects of the portal that motivated positive health behavior had to be actionable, meaning participation was encouraged to perform the activity, like scheduling an exam. Over 70% (17/24, 71%) of content had features that could motivate behavior. Every page within the “Preventive care you need now” section fulfilled the motivational behavior criteria by using verbs that promoted action, like “talk with your doctor” and “protect yourself.” Understanding that losing weight is challenging, on the “Your next steps” section of the “Diet” page, the text attempted to motivate patients by stating, “Keep eating five or more servings of vegetables and fruits per day.” However, we did not consider the “Smoking,” “Weight,” and “Flu vaccine” pages to be motivational because the content was overly passive. Confirming this analysis, patient #31, with diabetes, thought that the content should be “scarier” to motivate people to change their behavior. She acknowledged that losing weight would assist with her diabetes management, but after reviewing the content, she said, “I’m not scared enough to be doing all the things I ought to be doing.” Similarly, the “Flu” page recommended as a next step, “Talk with your doctor about getting a flu shot.” Although an action verb was used and previous content provided an overview of benefits, no urgency to get the vaccine was generated.


Interactivity included synchronous communication, the ability of a user to take control, or any function that enabled the user to become involved more deeply within the content. We found only 29% (7/24) of the pages to have characteristics of interactivity. There were very few opportunities to interact with the portal aside from reading text. Interactive quizzes, cholesterol calculators, and other risk assessment tools were presented as external links, which took the user away from the portal. Lack of interactivity was confirmed by a doctor who was frustrated over the system’s asynchronous communication. He said, “The patient can’t contact us directly. Instead, they have to go through the nurse and we review it a few hours later, or a day later, and then it becomes broken communication.” However, a relatively strong aspect of the portal’s interactivity was the ability for the user to control the navigation and choose the area accessed. For instance, the “Self-management tools” page allowed for patients to find specific information relevant to their condition, while the mammogram and cervical cancer pages enabled the patient to input dates of their last test and whether abnormal results appeared.

Opportunities for greater interactivity were available on most pages. For instance, the “Take aspirin” page asked questions about risk, but there was no way for the patient to answer those questions and receive timely feedback. Also, the “Your labs” page had a row of physician comments, but it was not possible for the patient to respond, make an appointment, or get more information.


Engagement received the lowest score with 8% (2/24). Only 2 webpages were deemed engaging, having content that made the user want to further explore the website by creating physical, cognitive, or emotional involvement.

The main “Preventive care you need now” page made bold proclamations, like “You have high blood pressure,” which would get a patient’s attention and impel them to further explore. The rest of the pages did not have enough personalized content to be engaging. For example, the “Mammogram” page made a recommendation for women age 40–84 years instead of providing more specific advice based on the patient’s specific age. The “Self-management tools” page included links within 19 specific content categories, including exercise, healthy diet, and smoking cessation, but required that a patient scroll through almost 200 links.

Consequences of Incidents and Immediacy Levels

We synthesized the patients’ and clinicians’ incidents with results of the content analysis to identify domains that would improve website functionality. We discovered the following 2 themes: (1) communication can be improved with directness and interactivity and (2) standardization contributes to patient disengagement.

Communication Can Be Improved With Directness and Interactivity

While the intention of the portal was to provide direct information to augment the patient’s care, both patients and clinicians found that the portal sometimes hampered the communication process. The portal provided general awareness overviews; however, patients yearned for specific information about how a particular treatment personally affected them. Therefore, patients used other online sources for more specific information.

Patient #9 said, “Sometimes I have back pain and I cannot come to the primary care doctor, but I need some information.” This patient’s comments aligned with many other patients’ views, whose first instinct was to seek outside sources rather than rely on the portal. Furthermore, patients needed detailed information between visits to manage chronic diseases. For instance, patient #31, with type 2 diabetes, was concerned about managing her blood sugar. She said, “I’m not sure I’m always doing the right things” and the portal could not be counted on as a resource to help her manage specificities of the disease. Patients became disinterested in the portal’s content, because as patient #19 said, “I pretty much know all this stuff.” Patient #21 considered the portal “dumbed down” and mentioned that, since he was due for a prostate exam, he already conducted basic research. Perhaps, if patients deemed content to be authoritative and felt that it was coming directly from the clinician, they would find it more helpful and seek out the portal between scheduled visits. By using alternative websites, patients would combine the vetted and personalized information from the portal with recommendations from the other websites, which may or may not be beneficial given the patient’s specific medical history.

Similarly, clinicians reported that they were unable to effectively communicate in an appropriate manner through the portal. They were unsure whether patients read their instructions, and asynchronous communication patterns disrupted care.

Standardization Contributes to Patient Disengagement

Although content was personalized, patients largely viewed the portal as just another platform offering standardized information, which minimized patients’ assessment of it as a resource for delivering personalized health recommendations.

Lack of Personal Relevance

Patient #25 became indifferent and dismissed the portal as a helpful tool after reading the section about weight loss. He responded, “Yeah, but what do I do with it? I’m overweight. I need to lose weight. Got it.” Patient #8 was unimpressed with the recommendations and said, “A lot of the information, the basics, this is stuff I could read on CNN health’s website. [It] doesn’t tell me much.” Similarly, patient #20 was disappointed with the portal’s lack of guidance about vitamins. Since the efficacy of vitamins is not evidence based and therefore would not be recommended by the clinician, the portal’s content did not address vitamins.

Not Motivational

Overall, the portal did not provide actionable information, leaving patients with little motivation to use the system. Patient #27 said, “This is not motivating. Maybe if it gave me examples, like go for a walk or have an apple with lunch.” Lack of specificity was best exemplified by patient #4, who visited the physician for a flu shot even though it had been years since she last received the vaccine. She thought the portal’s flu shot content was “generic” and it did not inspire her, but said, “I am here today because I’ve been told by my daughter’s baby’s pediatrician that I need to get a flu shot.”

Principal Findings

Conducting a multiple methodological study on a patient portal as a case study revealed that many factors need to be considered when delivering personalized health information via a portal. It was necessary to use multiple methods and involve numerous stakeholders to truly understand the complexity of the user experience. The combination of gathering critical incidents from patients and clinicians, and analyzing the themes of the portal’s content enabled us to simultaneously confirm the results through a concurrent triangulation strategy [43].

Findings suggest that personalized content must contain higher levels of engagement and interactivity. This may be accomplished through increased levels of immediacy, which has been associated with greater patient satisfaction, understanding [13,44], and compliance [45]. Communication strategies incorporating immediacy, like adding personalized content addressing patients’ concerns about treatment, can contribute to humanizing interactions between patients and clinicians, encouraging patient participation and building trust [8].

Making portals more immediate through other forms of health information technology hold the opportunity for greater levels of engagement [46]. For instance, interactive videos to inform patients about elective back surgery helped facilitate decision making and informed consent [47]. Compared with reading text, watching videos has the ability to improve patients’ knowledge of health risks [48]. Furthermore, videos featuring information directly from a patient’s clinician could make the content seem more authoritative and personal.

Although implementation and integration of health portals has initially been slow [49], dramatic growth is expected due to patient demand and physician adoption [50]. Therefore, it is necessary to design patient-centric systems that provide comprehensive, coordinated value to patients, while also meeting the needs of clinicians [51]. In fact, clinician endorsement and engagement with the portal is an important factor that influences patient adoption [52].

The portal used for this case study has been at the forefront of personalizing content, but even more is needed. While the majority of webpages fulfilled the definition of personalization, patients expected a higher level of personalization akin to sophisticated mobile phones and social networking websites. This tension between patients’ expectations of personalization and clinician guidelines is pronounced because patients’ expectations of the portal’s main function differ from clinicians’. While the goal of the portal used in this study was primarily to educate and motivate patients for preventive screenings, patients considered the portal to be a mechanism to organize all of their health care needs. Sections with the highest level of personalization were on the preventive care pages, but patients expected similar personalization levels on laboratory results and medication pages. Based on this study’s findings, the objective of portals needs to be better communicated to patients, and future modifications should consider patients’ expectations. A portal rich with immediacy not only has the capacity to fulfill patients’ desires, but also has the potential to improve the medical environment. Results from this study indicated that clinicians worry about increases in workload, but they also realize the potential for portals to create greater efficiency.

Limitations and Future Directions

There were several limitations to this study. It would have been beneficial to have greater representation of adults from various age ranges, ethnicities, and socioeconomic classes. Participants represented only a small subset of all users, and recruitment likely selectively targeted those with strongly positive or negative impressions. In addition, comparisons across other health care information systems would have been beneficial to contrast differences and illuminate the various ways that systems are used. Future research should focus on analyzing the content, design, and usability of portals. In addition, research should analyze how patients use the portal’s content to determine whether it is a factor influencing health behavior.


Through the analysis of this portal, we found that higher levels of immediacy are necessary to sufficiently motivate patients to take preventive care measures. Portals are becoming a fixture of every medical office; therefore, it is necessary to modernize portals so that they can achieve their potential of enhancing health communication and improving health outcomes. The methods used in this study can be replicated when analyzing other patient portals, and conclusions can inform designers to develop more effective health information communication.


This research was supported by grants from the National Cancer Institute (R01CA166375-01A1) and the National Center for Advancing Translational Sciences (UL1TR000058).

Conflicts of Interest

Virginia Commonwealth University holds the intellectual property rights to the interactive preventive care record evaluated in this study. Although the university and developers are entitled to the system’s revenue, MyPreventiveCare is a noncommercial product, and no revenues have been generated other than grant funding.

  1. Kreps GL. Achieving the promise of digital health information systems. J Public Health Res 2014 Dec 2;3(3):471 [FREE Full text] [CrossRef] [Medline]
  2. Pagliari C, Detmer D, Singleton P. Potential of electronic personal health records. BMJ 2007 Aug 18;335(7615):330-333 [FREE Full text] [CrossRef] [Medline]
  3. Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, et al. Barriers and facilitators to online portal use among patients and caregivers in a safety net health care system: a qualitative study. J Med Internet Res 2015;17(12):e275 [FREE Full text] [CrossRef] [Medline]
  4. Undem T. Consumers and Health Information Technology: A National Survey. Oakland, CA: California HealthCare Foundation; 2010 Apr 13.   URL: http:/​/www.​​~/​media/​MEDIA%20LIBRARY%20Files/​PDF/​PDF%20C/​PDF%20ConsumersHealthInfoTechnologyNationalSurvey.​pdf [accessed 2016-04-29] [WebCite Cache]
  5. Krist AH, Woolf SH, Rothemich SF, Johnson RE, Peele JE, Cunningham TD, et al. Interactive preventive health record to enhance delivery of recommended care: a randomized trial. Ann Fam Med 2012;10(4):312-319 [FREE Full text] [CrossRef] [Medline]
  6. Edsall RL, Adler KG. User satisfaction with EHRs: report of a survey of 422 family physicians. Fam Pract Manag 2008 Feb;15(2):25-32 [FREE Full text] [Medline]
  7. Lazard A, Watkins I, Mackert MS, Xie B, Stephens KK, Shalev H. Design simplicity influences patient portal use: the role of aesthetic evaluations for technology acceptance. J Am Med Inform Assoc 2016 Apr;23(e1):e157-e161. [CrossRef] [Medline]
  8. Kreps GL. Engaging health communication. In: Socha TJ, Pitts MJ, editors. The Positive Side of Interpersonal Communication. New York, NY: Routledge; 2012:249-258.
  9. Frymier A. A model of immediacy in the classroom. Commun Q 1994 Mar;42(2):133-144. [CrossRef]
  10. Henning Z. From barnyards to learning communities: student perceptions of teachers' immediacy behaviors. Qual Res Rep Commun 2012 Jan;13(1):37-43. [CrossRef]
  11. Egan G. Interpersonal Living: A Skills-Contract Approach to Human-Relations Training in Groups. Monterey, CA: Brooks/Cole Pub Co; 1976.
  12. Lee LA, Sbarra DA, Mason AE, Law RW. Attachment anxiety, verbal immediacy, and blood pressure: results from a laboratory-analogue study following marital separation. Pers Relatationships 2011 Jun;18(2):285-301 [FREE Full text] [CrossRef] [Medline]
  13. Richmond V, Smith RS, Heisel AD, McCroskey JC. Nonverbal immediacy in the physician/patient relationship. Commun Res Rep 2001 Jun;18(3):211-216. [CrossRef]
  14. Liu Y, Malin JL, Diamant AL, Thind A, Maly RC. Adherence to adjuvant hormone therapy in low-income women with breast cancer: the role of provider-patient communication. Breast Cancer Res Treat 2013 Feb;137(3):829-836 [FREE Full text] [CrossRef] [Medline]
  15. Kreps GL, Neuhauser L. Artificial intelligence and immediacy: designing health communication to personally engage consumers and providers. Patient Educ Couns 2013 Aug;92(2):205-210. [CrossRef] [Medline]
  16. Greenhalgh T, Hinder S, Stramer K, Bratan T, Russell J. Adoption, non-adoption, and abandonment of a personal electronic health record: case study of HealthSpace. BMJ 2010;341:c5814 [FREE Full text] [Medline]
  17. MyPreventiveCare: Helping You Take Care of Your Health. Richmond, VA: VCU Department of Family Medicine; 2016.   URL: [accessed 2016-04-29] [WebCite Cache]
  18. Krist A, Woolf S, Rothemich S, Longo D, Johnson R, Matzke G, et al. An Interactive Preventive Health Record (IPHR) to Promote Patient-Centered Preventive Care. 2011 Feb 28.   URL: https:/​/healthit.​​sites/​default/​files/​docs/​publication/​r18hs017046-krist-final-report-2011.​pdf [accessed 2016-04-27] [WebCite Cache]
  19. Creswell JW, Klassen AC, Plano Clark VL, Smith KC. Best Practices for Mixed Methods Research in the Health Sciences. Bethesda, MD: Office of Behavioral and Social Sciences Research; 2011 Aug.   URL: https:/​/obssr-archive.​​mixed_methods_research/​pdf/​Best_Practices_for_Mixed_Methods_Research.​pdf [accessed 2016-04-27] [WebCite Cache]
  20. Clark V, Creswell JW, Plano CVL. Understanding Research: A Consumer's Guide. Upper Saddle River, NJ: Merrill; 2010.
  21. Marshall MN. Sampling for qualitative research. Fam Pract 1996 Dec;13(6):522-525 [FREE Full text] [Medline]
  22. Tracy SJ. Qualitative Research Methods: Collecting Evidence, Crafting Analysis, Communicating Impact. Hoboken, NJ: Wiley-Blackwell; 2013.
  23. Flanagan JC. The critical incident technique. Psychol Bull 1954 Jul;51(4):327-358. [Medline]
  24. Woolsey LK. The critical incident technique: an innovative qualitative method of research. Can J Counselling 1986 Oct;20(4):242-254.
  25. Kemppainen JK. The critical incident technique and nursing care quality research. J Adv Nurs 2000 Nov;32(5):1264-1271. [Medline]
  26. Chell E. Critical incident technique. In: Cassell C, Symon G, editors. Qualitative Methods and Analysis in Organizational Research: A Practical Guide. London, UK: Sage Publications; 2004:45-60.
  27. Beech P, Norman IJ. Patients' perceptions of the quality of psychiatric nursing care: findings from a small-scale descriptive study. J Clin Nurs 1995 Mar;4(2):117-123. [Medline]
  28. Owen W. Interpretive themes in relational communication. Q J Speech 2009 Jun 05;70(3):274-287. [CrossRef]
  29. Charmaz K. Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis. London, UK: SAGE; 2006.
  30. Lincoln YS, Guba EG. Naturalistic Inquiry. Beverly Hills, CA: Sage Publications; 1985.
  31. Bartlett Ellis RJ, Carmon AF, Pike C. A review of immediacy and implications for provider-patient relationships to support medication management. Patient Prefer Adherence 2016;10:9-18 [FREE Full text] [CrossRef] [Medline]
  32. Hill CE, Gelso CJ, Chui H, Spangler PT, Hummel A, Huang T, et al. To be or not to be immediate with clients: the use and perceived effects of immediacy in psychodynamic/interpersonal psychotherapy. Psychother Res 2014;24(3):299-315. [CrossRef] [Medline]
  33. Rogers EM. Diffusion of Innovations. New York, NY: Free Press; 2003.
  34. Liu Y. Developing a scale to measure the interactivity of websites. J Advertising Res 2003 Aug 13;43(02):207-216.
  35. Wu D, Im II, Tremaine M, Instone K, Turoff M. A framework for classifying personalization scheme used on e-commerce websites. In: Proceedings of the 36th Annual Hawaii International Conference on System Sciences. 2003 Presented at: Annual Hawaii International Conference on System Sciences; 2003; Big Island, HI.
  36. Patterson P, Yu T, Ruyter KD. Understanding customer engagement in services. In: Advancing Theory, Maintaining Relevance. 2006 Presented at: ANZMAC 2006 conference; 2006; Brisbane, Australia.
  37. Calder BJ, Malthouse EC. Media engagement and advertising effectiveness. Kellogg Advertising Media 2008:1-36. [CrossRef]
  38. Higgins ET. Value from hedonic experience and engagement. Psychol Rev 2006 Jul;113(3):439-460. [CrossRef] [Medline]
  39. Fogg BJ. A behavior model for persuasive design. In: Proceedings of the 4th International Conference on Persuasive Technology. 2009 Presented at: International Conference on Persuasive Technology; 2009; Claremont, CA.
  40. Skemp RR. Relational understanding and instrumental understanding. Arithmetic Teacher 1978 Nov;26(3):9-15.
  41. Maxwell JA. Qualitative Research Design: An Interactive Approach (Applied Social Research Methods). Los Angeles, CA: Sage Publications; 2012.
  42. Fielding NG, Fielding JL. Linking Data: The Articulation of Qualitative and Quantitative Methods in Social Research. Beverly Hills, CA: Sage Publications; 1986:41-53.
  43. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage Publications; 2009.
  44. Larsen KM, Smith CK. Assessment of nonverbal communication in the patient-physician interview. J Fam Pract 1981 Mar;12(3):481-488. [Medline]
  45. Burgoon J, Pfau M, Parrott R, Birk T, Coker R, Burgoon M. Relational communication, satisfaction, compliance‐gaining strategies, and compliance in communication between physicians and patients. Commun Monogr 1987 Sep;54(3):307-324. [CrossRef]
  46. Ahern DK, Woods SS, Lightowler MC, Finley SW, Houston TK. Promise of and potential for patient-facing technologies to enable meaningful use. Am J Prev Med 2011 May;40(5 Suppl 2):S162-S172. [CrossRef] [Medline]
  47. Deyo RA, Cherkin DC, Weinstein J, Howe J, Ciol M, Mulley AG. Involving patients in clinical decisions: impact of an interactive video program on use of back surgery. Med Care 2000 Sep;38(9):959-969. [Medline]
  48. Steffenino G, Viada E, Marengo B, Canale R, Nursing and the Medical Staff of the Cardiac Catheterization Unit. Effectiveness of video-based patient information before percutaneous cardiac interventions. J Cardiovasc Med (Hagerstown) 2007 May;8(5):348-353. [CrossRef] [Medline]
  49. Rhodes HB. The PHR quandary: despite the benefits, issues of technology and trust slow adoption. J AHIMA 2007 Apr;78(4):66-7, 69. [Medline]
  50. EMR 2015: The Market for Electronic Medical Records. New York, NY: Kalorama Information; May 2015.
  51. Baird A, Nowak S. Why primary care practices should become digital health information hubs for their patients. BMC Fam Pract 2014;15:190 [FREE Full text] [CrossRef] [Medline]
  52. Irizarry T, DeVito Dabbs A, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015;17(6):e148 [FREE Full text] [CrossRef] [Medline]

CIT: critical incident technique

Edited by G Eysenbach; submitted 17.12.15; peer-reviewed by S Woods, T Irizarry; comments to author 03.04.16; revised version received 18.04.16; accepted 18.04.16; published 17.05.16


©Jordan M Alpert, Alex H Krist, Rebecca A Aycock, Gary L Kreps. Originally published in the Journal of Medical Internet Research (, 17.05.2016.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.