Consumers' Perceptions of Patient-Accessible Electronic Medical Records
Background: Electronic health information (eHealth) tools for patients, including patient-accessible electronic medical records (patient portals), are proliferating in health care delivery systems nationally. However, there has been very limited study of the perceived utility and functionality of portals, as well as limited assessment of these systems by vulnerable (low education level, racial/ethnic minority) consumers.
Objective: The objective of the study was to identify vulnerable consumers’ response to patient portals, their perceived utility and value, as well as their reactions to specific portal functions.
Methods: This qualitative study used 4 focus groups with 28 low education level, English-speaking consumers in June and July 2010, in New York City.
Results: Participants included 10 males and 18 females, ranging in age from 21-63 years; 19 non-Hispanic black, 7 Hispanic, 1 non-Hispanic White and 1 Other. None of the participants had higher than a high school level education, and 13 had less than a high school education. All participants had experience with computers and 26 used the Internet. Major themes were enhanced consumer engagement/patient empowerment, extending the doctor’s visit/enhancing communication with health care providers, literacy and health literacy factors, improved prevention and health maintenance, and privacy and security concerns. Consumers were also asked to comment on a number of key portal features. Consumers were most positive about features that increased convenience, such as making appointments and refilling prescriptions. Consumers raised concerns about a number of potential barriers to usage, such as complex language, complex visual layouts, and poor usability features.
Conclusions: Most consumers were enthusiastic about patient portals and perceived that they had great utility and value. Study findings suggest that for patient portals to be effective for all consumers, portals must be designed to be easy to read, visually engaging, and have user-friendly navigation.
J Med Internet Res 2013;15(8):e168)
A number of social and economic factors, such as rising health care costs, a trend towards home health care, as well as shortages of health care workers, have encouraged consumers to increasingly assume a more active role in the management of their own health. Concurrently, advisory bodies, such as the Institute of Medicine, and government agencies have promoted the ability of health information technology to not only enhance the patient-centeredness of health care, but to improve the quality and efficiency of health care delivery overall .
One proliferating electronic health (eHealth) tool is the patient-accessible electronic medical record or patient portal. A patient portal, as defined by HealthIT.gov, is an Internet application that allows patients to access their electronic health records and communicate with their health care providers . In this paper, we use patient portal to refer to a secure system tethered to a provider’s electronic medical record. The patient portal typically offers patients the ability to communicate with providers, manage medications, schedule appointments, review lab results, access medical history data, as well as provide links to obtain patient education or health information from other online sources [ ]. Patient portals allow consumers to take greater control of their health information by changing traditional top-down (doctor to patient) methods of health communication [ ] and improving satisfaction with provider communication and overall care [ ]. While research is limited, some studies suggest benefits such as greater engagement in health care with online access to personal health information [ ], improved rates of screening [ ], improved appointment adherence [ ], and greater sense of confidence and empowerment, increased knowledge about health, and improved health behaviors [ , ].
The Health Information Technology for Economic and Clinical Health (HITECH) Act has promoted the adoption of health information technology by incentivizing the meaningful use of electronic medical records. Tying incentive payments from Medicare and Medicaid to meeting meaningful use objectives, newly released requirements stipulate that patients must be given online access to their health record with the ability to access, print, share, or download their health information. In addition, providers will need to ensure that at least 5% of patients actively use this technology .
The study is motivated by no less than two factors. First, consumers are increasingly interested in accessing their personal health information online [- ]. The Markle Survey on Health in a Networked Life reports that roughly 70% of the public and 65% of doctors believe that patients should be able to download and keep their own health information [ ]. Walker and colleagues (2009) conducted focus groups with frequent Internet users and found that participants not only wanted electronic access to their medical records, but customized health information and advice as well [ ].
Second, the literature has documented the vulnerability of certain population groups to disparities in health outcomes [, ] and health care quality [ , ]. Vulnerable groups are most often described as racial and ethnic minorities, poor, under-educated, immigrants, and those lacking English proficiency [ ]. Many vulnerable populations are also likely to be low literate and/or low health literate [ , ]. Strong evidence exists linking low literacy and low health literacy with poorer health behaviors, disease management skills, and health outcomes [ - ]. This is particularly concerning considering most health information created for the general public is written at or above the 10th grade level [ ], with health websites often at an even higher level [ ], even though more than half of the adults in the United States read at an 8th grade level and lower, and some vulnerable populations read at 5th grade level or lower [ ].
Vulnerable populations continue to receive poor quality health care as well as face more barriers to receiving care than more advantaged groups, despite continuing efforts to reduce such disparities . Some have suggested that use of eHealth, by providing increased access to health information and support, may help to ameliorate disparities in vulnerable or disadvantaged groups [ , ]. However, there is conflicting data regarding these consumers’ perceptions and the use of patient portals and other electronic health information.
Studies with medically underserved and vulnerable patients [, ] found strong interest in accessing online health records, and research also shows that some vulnerable populations, such as racial and ethnic minorities, have similar or even greater interest in accessing health information online than national samples [ ]. Other studies show that racial and ethnic minorities have lower rates of enrollment [ - ], logging on [ , ], likelihood of receiving an access code, and regularly using a patient portal [ ] than non-minorities. However, once enrolled, studies found no disparities in use by race/ethnicity [ , ] or less of an association with race/ethnicity than with portal adoption [ ]. These findings indicate that the way patient portals are designed and presented to consumers may influence how portals are perceived, valued, and ultimately utilized.
Thus, our goal in this qualitative study was to identify what perceived utility and value vulnerable health consumers attach to the concept of patient portals and to core features and functions of these portals. We utilized focus groups targeting lower education level, minority residents of New York City to explore this question.
A total of 28 individuals, 10 male and 18 female, 21-63 years of age, with a high school education or less, participated in 4 focus groups held June-July 2010. Each of the 4 focus groups consisted of a convenience sample of 6-8 participants along with a moderator and a note taker. The groups were conducted using a semistructured format. Prior to commencing the focus groups, participants completed a technology experience questionnaire adapted from Czaja et al [, ]. Each group lasted approximately 90 minutes. All focus groups were audiotaped. At the end of each group, participants were paid US $40.00 and given a US $4.50 round-trip NYC MetroCard for their time and travel. All study protocols and materials were approved by the Mount Sinai School of Medicine Institutional Review Board.
Individuals were eligible to participate if they met the following criteria: New York City resident, aged 21-75 years, able to read and speak English, and no higher than a high school education/GED.
Recruitment was conducted at 3 sites in New York City: Mount Sinai Medical Center, East Harlem; Queens Library, Long Island City, Queens; and CAMBA, Flatbush, Brooklyn. These organizations and locations were chosen for their access to diverse population groups in different urban neighborhoods in New York City.
Each site is located in a health service area designated as Medically Underserved and a Primary Care Health Professional Shortage area by the United States Department of Health and Human Services Health Resources and Services Administration (HRSA). According to HRSA, Medically Underserved Areas have “too few primary care providers, high infant mortality, high poverty and/or high elderly population” .
A description of the focus group neighborhood, site, and participants can be found in[ - ].
We specifically recruited in Queens for 2 age groups: one 18-49 and the other, 50 and older. We chose this location because we believed they had the largest and most age-diverse pool of candidates meeting our criteria. Although there are a number of different age ranges within which researchers and others operationalize “younger” and “older” adults, we follow other studies and surveys, which have defined “older” adults as those 50 years and older [- ].
Using IRB (Institutional Review Board) approved recruitment and advertising materials, staff members of these programs assisted with recruitment of participants by distributing flyers about the study, including the inclusion criteria, to their participants and patrons. Interested individuals either signed a sheet requesting researchers contact them to be screened for the study or called researchers directly. Potential participants were screened by research staff over the phone to ensure they were eligible for the study. Thirty-two individuals met eligibility criteria, agreed to participate, and were scheduled to attend a focus group. Twenty-eight participants actually attended a group. All participants provided informed consent according to IRB requirements.
Procedure for Focus Groups
It is often the case in focus group research that participants are somewhat familiar with the topic of discussion. However, in this case, most participants were unfamiliar with the concept of patient portals and their specific capabilities. Therefore, the moderator provided a scripted introduction to patient portals including demonstrations using readily available informational videos as prompts. The videos were produced by a large health system, a managed care company, a multispecialty health care provider, and a federally qualified health center network—all of whom have been forerunners in the adoption of electronic medical records for patients. We selected clips from these organizations because they represented a broad range of possible entities from which patients may receive health care. The videos were created by these entities for their patients, so what participants saw was what a real world patient might encounter when being introduced to a portal for the first time.
The moderator’s guide focused on exploring the following four core functions of portals: scheduling appointments, managing medications, proxy functions, and reviewing lab results. These four functions were chosen because they are universally present in most patient portals. Participants were asked questions such as what they thought about the idea of portals in general and each of the four functions and how likely would they be to use the feature. Participants were also encouraged to ask questions and offer critiques of portal content and capabilities, as well as respond to a scenario that asked how a patient portal might be of use in keeping them healthy.
All focus groups were written up in quick notes by both the moderator and observer independently within 24 hours. Quick notes record initial impressions as these can extinguish rapidly over time. The analysis team, consisting of the moderator and note taker, established a process for analysis that consisted of developing a coding guide as follows. We used a grounded theory (GT) model to analyze focus group data [- ]. GT involves constantly comparing the data, coding, and identifying interchangeable indicators to reveal patterns that ultimately lead to categories. Two coders independently and repeatedly listened to the audio, referred to quick notes, and then conferred to discuss and refine emerging topics and themes. This process continued until the coding guide was finalized. The coding guide and accompanying narrative summary was developed to describe the key content of each group and enable a ready comparison of break characteristics, that is, the factors that differentiated one group from another.
All 28 participants had no higher than a high school degree, 15 had a high school diploma or GED, and 13 did not complete high school. Most were of ethnic/racial minority backgrounds, 7 Hispanic, 19 non-Hispanic black, 1 non-Hispanic white, and 1 Other, and all were of low economic means, including 8 with reported household incomes below US $20,000 (). Although more than half of participants refused or said that they did not know their annual household income, it is acknowledged that research participants may not provide income information because they are unsure of the answer or feel it is too private to share [ ]. However, research has also shown that those who refuse to provide income information are more likely to not be working, have less education, and live in a low socioeconomic neighborhood than those who report income [ ]. Given the neighborhood characteristics of our focus group locations, we are confident that our participants are representative of the vulnerable population we intended to reach.
Technology in People’s Daily Lives
First, we describe participants’ current technology and eHealth use, record-keeping behaviors, and health information seeking behaviors. Then, we discuss participants’ responses to selected features of patient portals. Finally, we present themes arising from participants’ attitudes and perceptions of patient portals.
The majority of participants currently used communication technology such as computers, the Internet, and mobile devices. All reported that they were currently using or had used a computer in the past and only 2 participants reported having no Internet experience (). All participants had either a basic cell phone or smart phone (some participants used both) ( ).
Most participants used a computer at home, but many also accessed computers in a variety of other places, such as at a public library, adult learning center, friend or relative’s house, work, or community center (). Many used technology to perform tasks of everyday life such as sending email, accessing social media, playing games, and searching for information online ( ). Most participants were not only technology users but were interested in learning more ways to stay connected through technology.
The East Harlem group was the most facile and “on the grid” of all of our participants. All 6 women regularly reported using a computer and the Internet in their daily lives. Facebook, email, pictures, Myspace, shopping, Twitter, school websites to check their children’s progress, planning travel/trips, or finding their way around town were among the things that they routinely do online. The Queens older adults group reported that they were learning how to access new features on their mobile devices or computers, with several expressing a desire to learn how to “text” and others regularly using social media. Members of the Brooklyn group were somewhat less likely to have used some forms of technology (the only group with some members reporting no Internet experience—see). However, they were more anxious to learn about and embrace technology than the Queens older adults, perhaps because they saw mastery of technology as a key part of their acculturation and social and economic success.
Previous use of technology for health-related issues was limited across all groups. Only 10 of 26 Internet-using participants reported ever having searched for health information online (), in contrast to national statistics showing 80% of all online adults have used the Internet to search for health information [ ]. However, there were a few participants who reported not only searching for health information, but accessing their own personal health information electronically. One participant (Queens 21-49) reported having access to a fully functioning patient portal, noting that she had a “health scare”, for which she regularly accessed a portal to help her “stay on top of my medical records”. Another participant in the Queens older adults group reported carrying his medical information on a jump drive attached to his keychain.
Most participants across groups recognized benefits in using technology and eHealth tools to manage their health care, although a few expressed concerns associated with these tools. Younger participants, such as those in the East Harlem group, were very comfortable with technology and saw the use of eHealth tools as an extension of what they were already doing in their everyday lives. They also expressed the most interest and excitement around features that would make their lives more “hassle free”, such as using the portal to make appointments and get copies of medical and immunization records for their children.
Likewise, participants accustomed to being online and using social media transferred their expectations for user-friendly formats and tools to the patient portals we showed them. Participants expressed a desire for features such as mouseovers/clicks for just-in-time information. They faulted a site for not having such common features. For some, having a website interface that resembled commonly used Internet sites affected how they felt toward the patient portals we were displaying. One participant summed this up by saying that if someone were trying to sell him on one of the systems we demonstrated, as an “average person”, he would pick the one that put “everything out front…it’s like working on Facebook or Hotmail, everything is in plain sight and I can deal with that and most people like that”.
We asked people about what kind of record keeper they would say they were, in order to compare this information to their level of enthusiasm about an electronic medical record. No strong connection was found. The groups differed in self-description of their general record-keeping behavior. Most participants, with the exception of the East Harlem group, asserted that they kept records in their daily lives. Some participants, especially those in the Queens older adults group, noted that they kept “paper records”, for things like tax purposes and appointments. Several participants across groups noted that they kept records of their health information: “I keep medical records, my copies of labs and test results from doctors”. Another participant, as noted earlier, kept his medical information electronically on a jump drive.
Sources of Health Information
When asked about sources of health information, participants reported using a mix of offline and online sources. Offline, participants reported going to the library, pharmacies, and health fairs, or, as one participant described “just talk[ing] to people who have a condition or a health care provider to get more information”. Of the online sources, WebMD and Google searches were most frequently mentioned as ways to obtain health information. However, although some participants commonly used the Internet to search for health information, several commented on the negative aspects of online health information. One participant described his experience with WebMD: “That thing is so hard to understand because they give you a schematic of a human body and you have to point to it but…it doesn’t really break it down where the average person can understand…some of the translation is in doctor terms, not in layman terms, so the average person that’s looking at it gets lost”. Others noted that Google searches result in “getting too much information” or finding searches “a little frustrating ’cause it’s not always exactly what you’re looking for, a specific answer to your problem”.
Initial Participant Response to the Idea of Patient Portals
After being given a short overview of patient portals, participants were asked to respond to the concept of a portal by rating the importance of access to a portal on a scale of 1-5 (with 1-Not Important to 5-Very Important). Most people across groups rated access to a portal either Very Important or Important to them, with the exception of the Queens older adults group. Participants in the East Harlem group embraced the concept from the outset, with all members rating potential access to a portal as “5-Very Important”, identifying benefits in the use of technology and patient portals to manage health care for both their children and themselves. These women envisioned the benefit in having remote access to test results, immunization records for children, and reminders for upcoming appointments (“I think it is great”). The majority of participants in the Brooklyn and Queens (21-49) groups rated patient portals highly, either a 4 or a 5. Members of the Queens older adults group had the least initial enthusiasm towards portals. Only 1 participant in this group rated access to a patient portal as “5-Very Important”, with 4 participants rating access to patient portals as “1-Not Important”.
Next, the groups were shown short videos demonstrating appointment setting, health proxy functions, medication management, and lab test results from a selection of patient portals.
Response to Patient Portals: “Housekeeping” Tasks
Almost all participants were impressed with the convenience and control portals provided for making appointments with their health care provider. Many reported frustration with the interactive voice response (IVR) systems they encountered when calling their providers and appreciated the convenience and control offered by a portal—“This seems to give more control over when you would like to make an appointment”. However, some participants found the “drop-down menu” a barrier to use. They were not sure that the appropriate options would be listed in such a menu or that the available choices would allow them to fully describe the nature or urgency of their complaint—“I would just be concerned about if it’s verified enough what you could put, the reasoning, that’s the only thing…when you speak with [a staff person] they might help you be more specific”. Thus, although participants appreciated the utility of the appointment-making feature, they still wanted to know that they had the option of calling their provider’s office directly.
Health Proxy Function
Many participants saw value in having proxy access to the electronic records of their children or their parents. Functions that would enable them to monitor their relative’s health, manage the medications their relative was taking, and understand more about how a relative’s chronic conditions should be managed on a regular basis were seen as especially valuable. In reference to caring for her elderly parent, one participant commented, “I’m dealing with my father who’s been having a stroke. I’m second health proxy and there’s three of us…[it is] better for communication”. Mothers in the East Harlem group felt the proxy function would increase convenience, especially if the portal allowed them to access health and vaccination records on the spot. Said one mother, “I have two kids and just with school and everything, they need physicals, immunizations, everything, if I can access that online, it would just cut out so much of the back and forth [participant in background, ‘hassle’], to actually have to go to your doctor’s office and wait for an hour just to get their records”.
Some consumers in the Queens 21-49 and Queens older adults groups were more circumspect about the possibility of an adult child or other family member gaining access: “Suppose you, as a parent, don’t want your child to know what is going on with you?” Continued discussion, however, assuaged some of these concerns, for example, understanding that the proxy is a voluntary act and that patients did not have to allow anyone, including their grown children, electronic access to their record.
When participants pondered the issue of parental access to an adolescent’s online health records, participants across all groups, except for the Brooklyn group, found privacy and confidentiality issues complicated to resolve. One participant believed that a parent having access to their teenager’s patient portal could inhibit the teen from talking to his or her doctor: “It might just make them more hesitant to disclose things to their doctor if they know that their parents can view this… it might just make them not want to say things that they don’t want their parents to know”. Another participant noted that even in current medical practice, “with teenagers of a certain age…they do have something where the parents are not allow[ed] to see certain things”. Thus, many participants believed a portal’s proxy function could be useful but acknowledged that rules concerning appropriate access needed further consideration.
Response to Patient Portals: Health Management Tasks
Use of a portal for medication management tasks, such as requesting medication refills, was not as popular a feature for participants. For some, the importance of the feature was not immediately apparent because, in the words of one participant, “I don’t get no medicine”. Many participants seemed more comfortable calling to get a prescription refilled or going directly to the pharmacy, noting that you may not always have access to a computer. However, the one participant who had access to a patient portal reported that she found this feature to be a great benefit, “With my doctor, if I need medication it goes straight to the pharmacy. It’s emailed right away. So it is excellent. Without having to wait for a paper prescription…By the time you get home you could pick your medicine up, so it’s very convenient”. Even those participants who did not feel that the medication management feature would be a benefit for them felt that it would be good for the “disabled and elderly”.
Lab and Test Results
Access to lab and test results is a central function of most patient portals. Most participants said being able to see their test results was a very important feature for them. All groups viewed the following example of a lab test result from a patient portal (typical of test result formats we have reviewed) ().
“Where is the standard?…The 0-7%, what exactly does that mean?” and “Is [that a] normal level for my blood?…I don’t know”. Participants wanted this page to be easier to use: “[be] more specific, like kind of in layman terms…Is that a good result? Is that a bad result?” or state clearly, “For a healthy person, it should be…you’re either above the window or below it”.
When participants were shown a test result that included a short explanatory note from the doctor (and ), they felt they better understood the test result numbers and whether they should be concerned or not, noting that, “It’s written. It’s clear. You can look at it and understand it. Sometimes the doctor’s visit is so fast you might not even get as much as that [in your visit]”. Another said, “A lot of times…the doctor explain[s] things, but he don’t explain things…99% of the time when people go to the doctor they want to get in and out…with this here, I truly feel because they broke [it] down. Okay, this is high but…you have a reminder, you have something…that’s there that you could look at.”
For some consumers, however, the accompanying doctor’s explanation did not go far enough. They wanted the test results page to explain the doctor’s recommendations, for example, that the cholesterol test results were not high enough to require medication and to change their diet, and a further explanation of the different test components and the risks associated with certain values in a way they could understand—the why behind the recommendation. Looking at the sample results for a cholesterol test, one participant commented, “See this, the numbers…[referring to the doctor’s note] he just telling me that other part, making me feel comfortable, but I want to know what’s that 210 and…what’s the difference between the LDL and the HDL…break it down for me”.
Consumer/Patient Empowerment—“Information is Power”
As reported above, most participants, with the exception of the Queens older adults group, were very interested in having access to their personal health information from the start. Many individuals expressed what can only be described as disbelief that the difficulties they historically have had accessing their medical records would be remedied. Participants recalled encountering cumbersome and daunting bureaucracies when they wanted a copy of their own medical records or that of a care recipient, such as a child or elderly parent (“Gotta go sign the paper, and you gotta wait a week, and then you gotta pick it up”). Others reported having to pay per page for copies of their medical record or were charged a co-pay to get test results.
Extending the Doctor’s Visit/Enhancing Communication With Health Care Providers
Participants almost unanimously named the short length of a doctor’s visit, the stress of not understanding things fully while at the visit, and the inability to access useful information post visit as characteristic of their health care experience. Patient portals were seen as offering at least a partial remedy for this very unsatisfactory situation. Many felt that online records could empower them with increased access to health information, resulting in a greater focus on their health and allowing them to be more proactive about taking care of their health.
The efficiency and portability of electronic medical records was seen as another huge benefit. Many participants said that their doctors used electronic medical records, and they felt that these records enhanced communication with providers: “When it was paper, when you go see a new doctor, they’re like asking you everything that you already given, all your demographics …when they can just look it up…and they’re not asking you all the same things all the time”. Although a few participants described themselves as “old school”, preferring in-person doctor-patient encounters without an electronic record (“the whole everything-on-the-computer thing is cool, it’s okay, but I’m kind of old school, I like the [personal]”) or accessing health records in paper format, there was also an acknowledgement that times change and with those changes would come greater acceptance of new technology (“For the next generation, it could be accepted…a lot of things that were thought about in the 60s is now being implemented in 2000”). Several participants, in fact, could foresee the benefits of enhanced communication with health care providers due to the portability of health information via patient portal: “I like it because if you go on vacation, you can get sick. And you have your medical record [and] you can give it to any doctor”. Another could see the utility of a portal to enable her to get more out of a doctor’s visit with her young children, “When I take my six-year-old to [an] appointment, I usually have my three-year-old there. They’re running around. You’re trying to listen to what the doctor’s telling you, but you’re still focused on, you know, your other kid. If you can go home and read everything…you’re more knowledgeable”.
Literacy and Health Literacy
Participants talked about the need to have patient portal information presented in ways “the average person can understand”. For example, after showing participants the sample lab test result page, one participant commented that she would use it “if I could read it and understand it…[but] I can’t understand it”. Another participant commented, “I just look and see numbers [referring to the sample results page]”. The reading and health literacy load (the demands on a reader’s understanding of science and other related concepts) of the sample lab test results in the demonstrated patient portal proved challenging and frustrating for participants. Participants found the reading and health literacy load of the content too high in 3 broad categories: (1) medical terms/medical information, eg, drug names, anatomy, chemistry, medical procedures (“There’s some people that don’t know the difference between good cholesterol and bad cholesterol. See, I didn’t know that until I got into my forties. Always just thought cholesterol. So that’s something they would have to explain”), (2) numeracy/numerical information, eg, number calculations, standard range, percentages (“Even if it does say…normal range, you’re still up by 1, your value, so that would kind of make some people freak out, like ok, what does that mean?”), and (3) design, navigation, and aiding tools, for which more than one participant suggested a design modification, such as an aiding tool that would allow you to click on the name of the test to get a more detailed explanation (“Could you click on each one and it can give you…a Wikipedia of it …?”).
Prevention and Health Maintenance
In the last segment of the focus groups, we asked participants to talk about what role portals might play in a user’s health. They commented that using the patient portal would allow people to “stay on top of [their] health”, “[focus on] prevention”, and “know more”. Although most were now positive about portals, they spoke in generalities about how a person’s health might improve as a result of using one. Most salient were the positive health impacts of reminders for appointments, annual visits, and screenings. Said one participant, “a lot of people don’t take charge of their health because they don’t even remember to take care of themselves, and a lot of times they don’t even know at what age they should be checking for what things”.
Privacy and Security Concerns
While privacy and security were not high priority concerns for most consumers we spoke with, in 3 of the 4 groups (with the exception of the Brooklyn group) there were a small number who voiced concerns related to the privacy and security of online records. An East Harlem participant, who asserted that access to a patient portal was very important, did so with the caveat: “If they are at your doctor’s office, you know anybody that has access to it, it’s like staff mainly, if it’s [a] website and…if there is some way someone can get your password…that’s a lot of information for someone else to have. Do you want to risk that?” A participant in the Queens 21-49 group feared that if someone was going through a divorce, an unhappy spouse might gain access and try to “damage the medical record” out of “spite”. Several members of the Queens older adults group feared “hacking” and identify theft; a couple of these participants initially felt that concerns over the privacy and security of electronic records would prohibit them from using a patient portal: “It comes down to a security point…the computer is awesome, I mean it’s off the chain. You could get on that computer, you could find out about anything, but do you really want your information up there?…I mean it’s a question”.
Some security and privacy concerns lessened once specific security features associated with patient portals were discussed, and participants identified the potential benefits of patient portals in their daily lives. Participants appeared willing to make tradeoffs, accepting the potential risk of breaches to their personal privacy for the convenience and accessibility of electronic records. Across groups, only one participant maintained throughout that privacy and security concerns would prevent her from using a patient portal, but she acknowledged portals should be an option for those who wanted them.
Response to the Concept of Patient Portals With More Information: Post Demonstration
After participants had the opportunity to view and discuss all of the demonstrations, they were asked to revisit their initial opinions of patient portals. Most participants, who were either indifferent or negative about the value of portals at the outset, raised their opinion of the importance of access to a portal once they had an opportunity to view and discuss common features; this was especially true in the Queens older adults group. Others, awakened to the possibilities, looked to the possible future benefits: “Is it possible, after let’s say, like your relatives, a parent or what have you, pass away, are you able to have access so that doctors can see what diseases and things that run in the family or…predisposition to be exposed to certain things?” A few wanted to know when a patient portal would be set up by their health care providers, so they could begin to access their records electronically.
Online technologies continue to change society and our daily lives in many ways. We see this graphically in the exponential growth in consumer use of the Internet and mobile devices to access health information and services [- ]. So too, the changes that patient electronic medical records portend are likely to change basic paradigms of patient provider interactions and the historic alignment of access to information.
Our goal in this study was to identify the perceived utility and value vulnerable health consumers attach to the concept of patient portals as well as to core features and functions of these portals: appointment setting, proxy functions, medication management, lab results, and preventive information. Our method was conducting a series of focus groups targeting lower education level, minority residents of New York City.
As expected, most consumers we spoke with were not familiar with a patient portal and only 1 participant out of 28 regularly used a portal. Despite this, most were positive about patient portals and positive perceptions increased over the course of the focus group discussion so that those less positive at the start, became more so as the groups evolved.
Our participants did not indicate that lack of Internet access or privacy when using public computers to access portals posed a potential barrier to portal use, as found elsewhere [, ]. Nor did our participants appear to have issues resulting from a lack of computer skills or experience, as has also been reported [ ]. This may be due the fact that all of our participants had a fair degree of prior experience with technology: all had previously used computers, almost all had experience using the Internet, and a majority had in-home Internet access.
As discussed in the Methods section, participants were introduced to a range of portal features and were asked to reflect on the utility and values of these features. Throughout the sessions, participants were engaged, asked frequent questions, and made extensive comments and critiques of specific features of portal features and functionality. Participants listened carefully to the descriptions and demonstrations of features, such as making appointments and reviewing lab tests. Then they posed their own questions, for example, “would you receive a confirmation of your appointment?”, “will the portal tell you whether that cholesterol number is ok?”, and, “if not, what you should do about it?” In contrast to perceptions by some consumers that it would be easy to learn how to use electronic medical records , participants in these groups identified a number of health literacy and usability barriers to patient portal use, such as complicated medical and numerical information, as well as a lack of aiding tools.
On a broader level, participants easily recognized the ability of electronic records to empower them through increased access to their own records and as a way to get more out of a doctor’s visit. Agarwal and colleagues found that patients who perceived electronic records as empowering had significantly higher intentions to use a portal . In our study, only a few expressed concern that electronic medical records would diminish their relationship with their provider. Instead, most participants focused on the benefits of portal use, especially in light of their increasing frustration with the ever shortening time with the doctor and growing complexity of the health information presented in visits. Research suggests that even if patients were able to understand all of the information related to their visit, only about half of the information would be remembered [ , ], if remembered correctly at all [ ]. Thus, introducing patient portals to these consumers allowed them to foresee using a portal as a tool to reduce the burden of remembering everything shared during a doctor’s visit by allowing access and retrieval of visit related information at their convenience. Changing perceptions so that more patients begin to view electronic medical records as an “extension” of the doctor’s visit and as a complementary tool, may ease concerns of those who worry that use of electronic records will supplant or depersonalize provider relationships.
The majority of participants judged their current methods of engaging with their health and “staying on top” of their health as in need of much improvement. Although participants offered up only a few tangible specifics, most perceived patient portals as a good tool for improving their knowledge and engagement in their health care and that of their loved ones.
As with all qualitative methods, focus groups are not representative of any larger group of people. It would be important and enlightening to conduct further qualitative study to see if these initial findings regarding interest, empowerment, most appealing functions, barriers to use, and other reactions hold up. Focus group participants did not readily introduce specific ways that their health behaviors might change as a result of using a patient portal. In part, this could be a product of the questions used in the moderator’s guide. We think it is likely that the artificial nature of the portal review in this setting did not provide enough specific and personalized content and context for participants to more substantively reflect on what they, as a real patient with their own health conditions, would most likely do with a patient portal.
The intent and promise of patient portals is that they will help engage people with their health, improve preventive care behaviors, and permit better management of chronic conditions. A handful of recent studies have begun to examine patients’ uptake of portals as well as patients’ perceptions and assessments of their actual use. Consistent with the views of our study participants, users report finding great value in patient portals , a perception shared by a wide range of patients, including those with mental health and substance abuse issues [ ] as well as patients with HIV [ ]. In general, patients report that portals positively impact communication with providers, and improved knowledge, empowerment, and self-care [ , ].
However, despite the positive potential of portals suggested by these findings, there still remains relatively little exploration of vulnerable patients’ experiences with these systems. In order to realize these benefits for all consumers, it is important to better understand the perceived and real barriers and opportunities that vulnerable groups, so disproportionately impacted by health disparities, face in actively using patient portals. This project moves a step beyond surveys and analyses about consumer use of patient portals by employing exploratory focus groups to study the perceptions and opinions of vulnerable, low educational level, and ethnic minority consumers in underserved communities. The findings signal that there is little reason to assume vulnerable populations are not accessing patient portals because they do not see the value of having electronic access to their personal health information. To the contrary, once consumers were given the opportunity to view and experience portal functions, they became very interested and motivated. Most participants perceived patient portals as a way to finally obtain information that has, until now, been out of their reach; as they often said, “knowledge is power.” We hypothesize that this target population, in bringing the experience and perspective that is a manifestation of their historically marginalized status, is, perhaps, even more motivated to embrace this and other emergent technologies.
These focus group findings serve as one baseline representation of public opinion regarding patient portals. Further study of public perception and preferences surrounding patient portals can be used to better guide the introduction of this technology to patients and add argument to the importance of promoting patient portal adoption and use. For example, some have suggested that increasing mobile access may encourage initial adoption and uptake of portals, since minorities are more likely to use these devices for health applications . In addition, our results demonstrate that portal content must be developed that accommodates the needs of lower education level, low literate, and low health literate users. Clearly, it is imperative that patient portals are designed and refined with the input of a diverse body of consumers in order to guard against continued barriers and poor access to information. If not, patient portals will remain an unused or underutilized tool for those who could potentially benefit the most.
The project described was supported by Award Number R21CA133487 from the National Cancer Institute. The manuscript is solely the responsibility of the authors and does not necessarily represent the official views of the National Cancer Institute or the National Institutes of Health.
Conflicts of Interest
- Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: Committee on Quality of Health Care in America, National Academy Press; 2001.
- HealthIt.gov. What is a patient portal?. URL: http://www.healthit.gov/providers-professionals/frequently-asked-questions [accessed 2013-03-17] [WebCite Cache]
- Shaw RJ, Ferranti J. Patient-provider internet portals--patient outcomes and use. Comput Inform Nurs 2011 Dec;29(12):714-8; quiz 719. [CrossRef] [Medline]
- Kreps GL, Neuhauser L. New directions in eHealth communication: opportunities and challenges. Patient Educ Couns 2010 Mar;78(3):329-336. [CrossRef] [Medline]
- Lin CT, Wittevrongel L, Moore L, Beaty BL, Ross SE. An Internet-based patient-provider communication system: randomized controlled trial. J Med Internet Res 2005;7(4):e47 [FREE Full text] [CrossRef] [Medline]
- Emont S. Measuring the impact of patient portals: What the literature tells us. 2011. URL: http://www.chcf.org/~/media/MEDIA%20LIBRARY%20Files/PDF/M/PDF%20MeasuringImpactPatientPortals.pdf [accessed 2013-06-11] [WebCite Cache]
- Leveille SG, Huang A, Tsai SB, Weingart SN, Iezzoni LI. Screening for chronic conditions using a patient internet portal: recruitment for an internet-based primary care intervention. J Gen Intern Med 2008 Apr;23(4):472-475 [FREE Full text] [CrossRef] [Medline]
- Horvath M, Levy J, L'Engle P, Carlson B, Ahmad A, Ferranti J. Impact of health portal enrollment with email reminders on adherence to clinic appointments: a pilot study. J Med Internet Res 2011;13(2):e41 [FREE Full text] [CrossRef] [Medline]
- Akesson KM, Saveman BI, Nilsson G. Health care consumers' experiences of information communication technology--a summary of literature. Int J Med Inform 2007 Sep;76(9):633-645. [CrossRef] [Medline]
- Neuhauser L, Kreps GL. eHealth communication and behavior change: promise and performance. Social Semiotics 2010 Feb;20(1):9-27. [CrossRef]
- American Med Association. Will meaningful use spur the growth of patient portals?. URL: http://www.ama-assn.org/amednews/2012/09/17/bisb0917.htm [accessed 2012-11-16] [WebCite Cache]
- Markle Foundation Initiative. Connecting for health - Americans overwhelmingly believe electronic personal health records could improve their health. URL: http://www.policyarchive.org/handle/10207/bitstreams/15533.pdf [accessed 2012-11-16] [WebCite Cache]
- California HealthCare Foundation. Consumers and Health Information Technology: A National Survey. 2010. URL: http://www.chcf.org/~/media/MEDIA%20LIBRARY%20Files/PDF/C/PDF%20ConsumersHealthInfoTechnologyNationalSurvey.pdf [accessed 2012-11-16] [WebCite Cache]
- Markle Foundation. Survey Finds Americans Want Electronic Personal Health Information to Improve Own Healthcare. URL: http://www.policyarchive.org/handle/10207/bitstreams/15500.pdf [accessed 2012-11-16] [WebCite Cache]
- Deloitte Center for Health Solutions. Survey of Health Care Consumers: Key Findings, Strategic Implications. 2009. URL: http://www.deloitte.com/view/en_US/us/Insights/centers/center-for-health-solutions/5735e23a4b101210VgnVCM100000ba42f00aRCRD.htm [accessed 2012-11-16] [WebCite Cache]
- Ross SE, Todd J, Moore LA, Beaty BL, Wittevrongel L, Lin CT. Expectations of patients and physicians regarding patient-accessible medical records. J Med Internet Res 2005;7(2):e13 [FREE Full text] [CrossRef] [Medline]
- Markle Foundation. Markle Survey on Health in a Networked Life, Snapshot: The Public and Doctors Agree with Blue Button Idea. URL: http://www.markle.org/sites/default/files/20101007_bluebutton_summary_1.pdf [accessed 2012-11-16] [WebCite Cache]
- Walker J, Ahern DK, Le LX, Delbanco T. Insights for internists: "I want the computer to know who I am". J Gen Intern Med 2009 Jun;24(6):727-732 [FREE Full text] [CrossRef] [Medline]
- Adler NE, Stewart J. Health disparities across the lifespan: meaning, methods, and mechanisms. Ann N Y Acad Sci 2010 Feb;1186:5-23. [CrossRef] [Medline]
- Smedley BD. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. Washington, DC: Board of Health Sciences Policy, Institute of Medicine; 2003.
- Hussey P, Anderson G, Berthelot JM, Feek C, Kelley E, Osborn R, et al. Trends in socioeconomic disparities in health care quality in four countries. Int J Qual Health Care 2008 Feb;20(1):53-61 [FREE Full text] [CrossRef] [Medline]
- AHRQ. National Healthcare Disparities Report. 2011. URL: http://www.ahrq.gov/research/findings/nhqrdr/nhdr11/index.html [accessed 2013-03-17] [WebCite Cache]
- Roundtable on Health Literacy - Institute of Medicine. Innovations in Health Literacy: Workshop Summary. Washington, DC: The National Academies Press; 2011.
- Kutner M, Jin Y, Paulsen C. The health literacy of America's adults: Results from the 2003 National Assessment of Adult Literacy. Washington DC: National Center for Education Statistics; 2006. URL: http://nces.ed.gov/pubs2006/2006483_1.pdf [accessed 2013-03-17] [WebCite Cache]
- Rudd RE. Health literacy skills of U.S. adults. Am J Health Behav 2007;31 Suppl 1:S8-18. [CrossRef] [Medline]
- Nielsen-Bohlman L, Panzer AM, Kindig DA. Health Literacy: A Prescription to End Confusion. Washington, DC: Institute of Medicine of the National Academies; 2004.
- Schwartzberg JL, Van Geest JC, Wang CC. Understanding Health Literacy: Implications For Medicine And Public Health. Chicago: American Medical Association; 2005.
- Dewalt DA, Berkman ND, Sheridan S, Lohr KN, Pignone MP. Literacy and health outcomes: a systematic review of the literature. J Gen Intern Med 2004 Dec;19(12):1228-1239 [FREE Full text] [CrossRef] [Medline]
- Berkman N, Donahue KE, Halpern DJ, Viera A, Crotty K, Holland A, et al. Health literacy interventions and outcomes: An updated systematic review. Evidence Report/Technology Assesment No. 199. (Prepared by RTI International–University of North Carolina Evidencebased Practice Center under contract No. 290-2007-10056-I. AHRQ Publication Number 11-E006. Rockville, MD: Agency for Healthcare Research and Quality; 2011. URL: http://www.ahrq.gov/legacy/clinic/epcsums/litupsum.htm [accessed 2013-03-17] [WebCite Cache]
- Baker DW, Gazmararian JA, Williams MV, Scott T, Parker RM, Green D, et al. Functional health literacy and the risk of hospital admission among Medicare managed care enrollees. Am J Public Health 2002 Aug;92(8):1278-1283. [Medline]
- Davis TC, Williams MV, Marin E, Parker RM, Glass J. Health literacy and cancer communication. CA Cancer J Clin 2002;52(3):134-149. [Medline]
- Stableford S, Mettger W. Plain language: a strategic response to the health literacy challenge. J Public Health Policy 2007;28(1):71-93. [CrossRef] [Medline]
- Zarcadoolas C, Blanco M, Boyer JF, Pleasant A. Unweaving the Web: an exploratory study of low-literate adults' navigation skills on the World Wide Web. J Health Commun 2002;7(4):309-324. [CrossRef] [Medline]
- Kirsch I, Jungeblut A, Jenkins L, Kolstad A. Adult Literacy in America: A First Look at the Results of the National Adult Literacy Survey (NALS). Washington, DC: National Center for Education Statistics, U.S. Department of Education; 1993.
- Chang BL, Bakken S, Brown SS, Houston TK, Kreps GL, Kukafka R, et al. Bridging the digital divide: reaching vulnerable populations. J Am Med Inform Assoc 2004;11(6):448-457 [FREE Full text] [CrossRef] [Medline]
- Dhanireddy S, Walker J, Reisch L, Oster N, Delbanco T, Elmore JG. The urban underserved: attitudes towards gaining full access to electronic medical records. Health Expect 2012 Jun 28:-. [CrossRef] [Medline]
- Schickedanz A, Huang D, Lopez A, Cheung E, Lyles CR, Bodenheimer T, et al. Access, interest, and attitudes toward electronic communication for health care among patients in the medical safety net. J Gen Intern Med 2013 Jul;28(7):914-920. [CrossRef] [Medline]
- Patel VN, Dhopeshwarkar RV, Edwards A, Barron Y, Likourezos A, Burd L, et al. Low-income, ethnically diverse consumers' perspective on health information exchange and personal health records. Inform Health Soc Care 2011 Dec;36(4):233-252. [CrossRef] [Medline]
- Goel MS, Brown TL, Williams A, Hasnain-Wynia R, Thompson JA, Baker DW. Disparities in enrollment and use of an electronic patient portal. J Gen Intern Med 2011 Oct;26(10):1112-1116 [FREE Full text] [CrossRef] [Medline]
- Roblin DW, Houston TK, Allison JJ, Joski PJ, Becker ER. Disparities in use of a personal health record in a managed care organization. J Am Med Inform Assoc 2009;16(5):683-689 [FREE Full text] [CrossRef] [Medline]
- Yamin CK, Emani S, Williams DH, Lipsitz SR, Karson AS, Wald JS, et al. The digital divide in adoption and use of a personal health record. Arch Intern Med 2011 Mar 28;171(6):568-574. [CrossRef] [Medline]
- Sarkar U, Karter AJ, Liu JY, Adler NE, Nguyen R, López A, et al. Social disparities in internet patient portal use in diabetes: evidence that the digital divide extends beyond access. J Am Med Inform Assoc 2011 May 1;18(3):318-321 [FREE Full text] [CrossRef] [Medline]
- Lyles CR, Harris LT, Jordan L, Grothaus L, Wehnes L, Reid RJ, et al. Patient race/ethnicity and shared medical record use among diabetes patients. Med Care 2012 May;50(5):434-440. [CrossRef] [Medline]
- Ancker JS, Barrón Y, Rockoff ML, Hauser D, Pichardo M, Szerencsy A, et al. Use of an electronic patient portal among disadvantaged populations. J Gen Intern Med 2011 Oct;26(10):1117-1123 [FREE Full text] [CrossRef] [Medline]
- Gustafson DH, Hawkins RP, Boberg EW, McTavish F, Owens B, Wise M, et al. CHESS: 10 years of research and development in consumer health informatics for broad populations, including the underserved. Int J Med Inform 2002 Nov 12;65(3):169-177. [Medline]
- Czaja S, Charness N, Dijkstra K, Fisk A, Roger W, Sharit J. CREATE Common Core Battery measures. (CREATE-2006-01) .
- Czaja SJ, Charness N, Fisk AD, Hertzog C, Nair SN, Rogers WA, et al. Factors predicting the use of technology: findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE). Psychol Aging 2006 Jun;21(2):333-352 [FREE Full text] [CrossRef] [Medline]
- US Department of Health and Human Services - Health Resources Services Administration. Find Shortage Areas: MUA/P by State and County. URL: http://muafind.hrsa.gov/ [accessed 2013-03-17] [WebCite Cache]
- Olson E, VanWye G, Kerker B, Thorpe L, Frieden T. Take Care East Harlem. NYC Community Health Profiles 2006;21(42):1-16 [FREE Full text] [WebCite Cache]
- Olson E, VanWye G, Kerker B, Thorpe L, Frieden T. Take Care Northwest Queens. NYC Community Health Profiles 2006;29(42):1-16 [FREE Full text] [WebCite Cache]
- US Census Bureau. 2010 Census, SF1, Population Division - New York City Department of City Planning. URL: http://maps.nyc.gov/census/ [accessed 2013-07-29] [WebCite Cache]
- Olson E, VanWye G, Kerker B, Thorpe L, Frieden T. Take Care Flatbush. NYC Community Health Profiles 2006;14(42):1-16 [FREE Full text] [WebCite Cache]
- Office of Institutional Advancement and the Office of Planning SDMC. Report on Brooklyn Community Health. 2007. URL: http://www.downstate.edu/bhr/reports/Community-Health-Report.pdf [accessed 2013-03-17] [WebCite Cache]
- Abrahams JP, Hoyer WJ, Elias MF, Bradigan B. Gerontological research in psychology published in the Journal of Gerontology 1963-1974: Perspectives and progress. J Gerontol 1975 Nov;30(6):668-673. [Medline]
- Killoran A, Dalley G. Promoting the health of older people: a compendium.: Health Education Authority; 1997. URL: http://www.gserve.nice.org.uk/nicemedia/documents/promohealth_older_comp.pdf [accessed 2013-03-17] [WebCite Cache]
- Lake Research Partners. National Survey of Adults 50 and Older on Experiences with Health Care System. URL: http://www.nationalpartnership.org/site/DocServer/Lake_Poll_Media_Report_FINAL.pdf?docID=6242 [accessed 2013-03-17] [WebCite Cache]
- Glaser B. The constant comparative method of qualitative analysis. Social Problems 1965;12:436-445.
- Glaser B. Basics of Grounded Theory Analysis: Emergence Vs. Forcing. Mill Valley, CA: Sociology Press; 1992.
- Holton J. The coding process and its challenges. In: Bryant A, Charmaz K, editors. Sage Handbook of Grounded Theory. London: Sage Publications; 2007:265-289.
- Kim S, Egerter S, Cubbin C, Takahashi ER, Braveman P. Potential implications of missing income data in population-based surveys: an example from a postpartum survey in California. Public Health Rep 2007;122(6):753-763 [FREE Full text] [Medline]
- Winkleby MA, Cubbin C. Influence of individual and neighbourhood socioeconomic status on mortality among black, Mexican-American, and white women and men in the United States. J Epidemiol Community Health 2003 Jun;57(6):444-452 [FREE Full text] [Medline]
- Fox S. Health topics. American: Pew Internet & Life Project; 2011. URL: http://www.pewinternet.org/Reports/2011/HealthTopics.aspx [accessed 2012-11-16] [WebCite Cache]
- Vance K, Howe W, Dellavalle RP. Social internet sites as a source of public health information. Dermatol Clin 2009 Apr;27(2):133-6, vi. [CrossRef] [Medline]
- Chou WY, Hunt YM, Beckjord EB, Moser RP, Hesse BW. Social media use in the United States: implications for health communication. J Med Internet Res 2009;11(4):e48 [FREE Full text] [CrossRef] [Medline]
- Fox S, Duggan M. Mobile Health: Half of smartphone owners use their devices to get health information and one-fifth of smartphone owners have health apps.: Pew Internet Research; 2012. URL: http://pewinternet.org/~/media//Files/Reports/2012/PIP_MobileHealth2012.pdf [accessed 2013-05-31] [WebCite Cache]
- Purcell K, Entner R, Henderson N. The Rise of Apps Culture. 2010. URL: http://pewinternet.org/Reports/2010/The-Rise-of-Apps-Culture.aspx [accessed 2013-05-31] [WebCite Cache]
- Luque AE, van Keken A, Winters P, Keefer MC, Sanders M, Fiscella K. Barriers and Facilitators of Online Patient Portals to Personal Health Records Among Persons Living With HIV: Formative Research. JMIR Res Protoc 2013;2(1):e8. [CrossRef] [Medline]
- Hilton JF, Barkoff L, Chang O, Halperin L, Ratanawongsa N, Sarkar U, et al. A cross-sectional study of barriers to personal health record use among patients attending a safety-net clinic. PLoS One 2012;7(2):e31888 [FREE Full text] [CrossRef] [Medline]
- Agarwal R, Anderson C, Zarate J, Ward C. If we offer it, will they accept? Factors affecting patient use intentions of personal health records and secure messaging. J Med Internet Res 2013;15(2):e43 [FREE Full text] [CrossRef] [Medline]
- Joyce CR, Caple G, Mason M, Reynolds E, Mathews JA. Quantitative study of doctor-patient communication. Q J Med 1969 Apr;38(150):183-194. [Medline]
- Roter DL, Hall JA. Studies of doctor-patient interaction. Annu Rev Public Health 1989;10:163-180. [CrossRef] [Medline]
- Anderson JL, Dodman S, Kopelman M, Fleming A. Patient information recall in a rheumatology clinic. Rheumatol Rehabil 1979 Feb;18(1):18-22. [Medline]
- Woods SS, Schwartz E, Tuepker A, Press NA, Nazi KM, Turvey CL, et al. Patient experiences with full electronic access to health records and clinical notes through the My HealtheVet Personal Health Record Pilot: qualitative study. J Med Internet Res 2013;15(3):e65 [FREE Full text] [CrossRef] [Medline]
- Urowitz S, Wiljer D, Dupak K, Kuehner Z, Leonard K, Lovrics E, et al. Improving diabetes management with a patient portal: a qualitative study of diabetes self-management portal. J Med Internet Res 2012;14(6):e158 [FREE Full text] [CrossRef] [Medline]
|CAMBA: a community-based organization in Brooklyn committed to serving low-income, homeless, and immigrant populations, among other vulnerable groups.|
|GT: grounded theory|
|HITECH: Health Information Technology for Economic and Clinical Health Act|
|HRSA: Health Resources and Services Administration|
|IRB: Institutional Review Board|
Edited by G Eysenbach; submitted 20.12.12; peer-reviewed by YM Tan, A Hörbst; comments to author 20.01.13; revised version received 18.03.13; accepted 13.06.13; published 26.08.13
©Christina Zarcadoolas, Wendy L Vaughon, Sara J Czaja, Joslyn Levy, Maxine L Rockoff. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 26.08.2013.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.