Background: Health care portals have the potential to provide consumers with timely, transparent access to health care information and engage them in the care process.
Objective: The objective was to examine the use, utility, and impact on engagement in care and caregiver-provider communication of a client/family portal providing access to electronic health records (EHRs) and secure, 2-way e-messaging with care providers.
Methods: We conducted a prospective, mixed-methods study involving collection of caregivers’ portal usage information over a 14-month period (from portal introduction in January 2015 to the end of the study period in March 2016), a Web-based survey for caregivers administered after a minimum of 2 months’ exposure to the portal and repeated 2 months later, and focus groups or individual interviews held with caregivers and service providers at the same points in time. The survey assessed caregivers’ perceptions of the utility of and satisfaction with the EHR and e-messaging, and the portal’s impact on client engagement and perceptions of caregiver-provider communication. A total of 18 caregivers (parents) completed surveys and 6 also took part in focus groups or interviews. In addition, 5 service providers from different disciplines took part in focus groups or interviews.
Results: Although usage patterns varied, the typical pattern was a steady level of use (2.5 times a month over an average of 9 months), which is higher than typically reported use. The portal pages most frequently accessed were the home page, health record main page, appointment main page, and reports main page. The Web-based survey captured caregivers’ perceptions of usefulness of and satisfaction with the EHR and portal messaging, as well as the portal’s impact on their engagement in care and perceptions of caregiver-provider communication. The surveys indicated a moderate degree of utility of and satisfaction with the portal features, and a low but emerging impact on engagement in care and caregiver-provider communication (survey scales measuring these outcomes displayed excellent internal consistency, with Cronbach alpha ranging from .89 to .95). Qualitative themes from focus groups and interviews supported and extended the survey findings. Caregivers and service providers saw appreciable information benefits and provided recommendations to increase portal use and utility. Caregivers focused on the scope of organizational adoption of the portal system and indicated their hopes for the future of the portal, whereas service providers were concerned about how to best manage their investment of time and effort in preparing client-friendly reports and messaging clients via the portal.
Conclusions: Overall, the findings show the promise of the portal and the need for ongoing evaluation to show the portal’s ultimate potential in enhancing engagement in care and communication with care providers.
Health care portals have the potential to improve consumers’ access to information, engagement in care, and health outcomes . Few children’s hospitals in North America offer online portals to families [ ] and, to our knowledge, there are no published research studies of portal use, utility, or impact in pediatric rehabilitation hospitals. Through electronic access to health records and e-messaging (secure 2-way messaging), clients and families may feel a greater personal connection to care and partnership in the care process.
In pediatric rehabilitation, the chronic nature of children’s difficulties and the need for parental support have resulted in widespread adoption of family-centered care, which stresses the importance of engaging in partnerships with care providers and exchanging information . Engagement in care refers to a sense of meaningful involvement, true collaboration, or authentic participation [ ], and the active involvement and full investment of children and parents in therapy is presumed to be essential to achieving positive client outcomes [ , ]. Similarly, good client-provider communication is considered essential in the delivery of children’s rehabilitation services [ ]. Effective communication is strongly associated with client satisfaction [ , ], and studies have indicated that communication plays an important role in the ongoing client-practitioner relationship, and especially in engaging clients [ ].
The Promise of Health Care Portals
Consumers are asking for the ability to view their appointment schedules online, have electronic access to documentation and other resources, and have their questions answered over the Internet. E-scheduling, e-visits, and e-messaging are seen as important portal functions , as they allow information exchange and can potentially enhance consumers’ ability to manage their own health and health care [ ]. By having access to a portal that does more than just provide information and data, consumers can become more active participants in their own health care [ ]. Thus, portals have the potential to provide more personalized, collaborative, and effective care. There is, however, limited (but growing) research information on the use of electronic communication in health care, including electronic health records (EHRs) [ , ].
Research on EHRs has typically focused on adult medical care and management of chronic diseases such as rheumatoid arthritis  and diabetes [ ] rather than children’s chronic conditions [ ]. We searched for articles on portals in pediatric rehabilitation hospitals (using “pediatric” or “paediatric” with “online” and “portal,” with searches limited to 2015 and more recent). We found articles on portals for children with mental health conditions [ ], with asthma [ ], and those receiving acute care in a pediatric hospital [ ]. We found no articles on portals in pediatric rehabilitation hospitals. A recent evaluation of a portal at a Canadian children’s acute care hospital indicated that, at 3 months after EHR implementation, service providers perceived a negative impact on their efficiency, productivity, and the quality of care they provided, whereas patients and families did not perceive a negative impact on care [ ].
Even for adult chronic disease management, there is sparse evidence to support the ability of personal health records and EHRs to improve efficiencies, reduce costs, and improve outcomes [, ]. Most existing research on health care portals has examined the frequency of use of their features, satisfaction with the portal, or efficiencies in health care utilization (eg, number of office and emergency department visits, phone calls) [ - ].
Systematic reviews on the effects of portals with EHR access [- ] indicate that few studies have investigated outcomes concerning patient engagement or empowerment, with studies typically providing nonsignificant findings. Kruse et al [ ] found only 27 studies of relevance to impact—those measuring meaningful outcomes such as patient participation in medical decisions, patient-provider communication, or satisfaction. For example, a study by van der Vaart and colleagues [ ] found that approximately 44% of patients with rheumatoid arthritis reported feeling more involved in treatment as a result of access to their EHRs; however, significant pre-post differences in empowerment were not found. Thus, research is needed to substantiate “meaningful use” impacts [ , ].
Summary of Research Gaps
To our knowledge, there has been no research on client portals for pediatric rehabilitation service users. As well, the literature has focused on portal use, satisfaction, and service utilization outcomes, and there is a recognized need to examine meaningful outcomes such as client engagement in care and communication with providers. These 2 outcomes are of fundamental importance in pediatric rehabilitation, as well as in medical services for people with chronic conditions . Furthermore, although some studies have tracked portal use over time (eg, [ ]), studies have not examined portal impact prospectively. Examining portal use, utility, and impact over time provides an opportunity to see how users access portal features and may also indicate emergent short-term impacts [ ], such as enhanced engagement in care. Since studies typically evaluate portal features in isolation [ ], there is benefit in examining client perceptions of the introduction of a full portal over a period of time. Lastly, the literature indicates the value of mixed-methods studies, which are rare in portal research [ ].
The Client Portal at Holland Bloorview Kids Rehabilitation Hospital
Holland Bloorview Kids Rehabilitation Hospital (Holland Bloorview) in Toronto, Ontario, is Canada’s largest children’s rehabilitation hospital. The hospital is a provincial resource for children with cerebral palsy, acquired brain injury, muscular dystrophy, amputation, epilepsy, spina bifida, arthritis, cleft lip and palate, autism, and other physical and developmental disabilities. Holland Bloorview’s vision is to create a world of possibility for kids with disability by embracing client- and family-centered care, and participating in applied research and education.
In early 2015, Holland Bloorview launched a consumer portal with the ultimate goal of helping clients and their caregivers (family members, most typically parents) take an active role in managing their own or their children’s care. The consumer health portal (called connect2care) was developed in partnership with clients and families . Connect2care provides clients and families with electronic access to their medical records, online appointment cancelling and booking features, transparent and timely access to clinical documentation, and e-messaging to connect with their care providers.
Beginning in January 2015, enrollment to the portal was initiated, first for clients on the inpatient units, and then also for ambulatory and community programs. In this first phase, the functions available included the ability to view the client’s schedule and visit history, view and print clinical notes, and update demographic details. In May 2015, improvements were made to the viewing of laboratory and microbiology test results, and new processes were established to increase the number of clinical notes that would flow to the portal (allowing providers from additional health disciplines to share clinical notes). Over the summer of 2015, training of more than 100 health care providers was completed to support this enhanced sharing of clinical notes. In the summer and fall of 2015, messaging functionality was rolled out, allowing secure e-messaging between portal users and their providers. By December 2015, the training of providers for e-messaging was completed.
Enrollment and usage targets for 2015 were established prior to launching the portal, with a target enrollment of 721 users (clients and caregivers), and a target use (unique logins) of 1440. The targets were met well ahead of schedule and, by the end of 2015, there were 869 enrolled users and more than 4800 uses. The adoption rate was approximately 12.41% (869 out of about 7000 unique clients). As well, over 200 staff were live with e-messaging at the end of 2015, including physicians, occupational therapists, physical therapists, speech-language pathologists, social workers, psychologists, therapeutic recreation staff, orthotics and prosthetics staff, ambulatory care nurses, child life workers, and nurse practitioners.
Study Purpose and Objectives
The purpose of our study was to examine the use, utility, and impact of the connect2care portal from the beginning of portal introduction until the end of data collection 14 months later. To meet this aim, we conducted a prospective, mixed-methods study collecting quantitative survey data and qualitative data from focus groups and interviews with caregivers and service providers at 2 points in time. We adopted a concurrent triangulation approach, in which quantitative and qualitative data were collected at the same time  and integrated at the level of interpretation [ ]. Our research objectives were (1) to determine caregivers’ portal use over the study period, (2) to examine levels of perceived usefulness of and satisfaction with the portal’s EHR and e-messaging, as well as the portal’s impact on engagement in care and caregiver-provider communication, and (3) to ascertain caregivers’ and service providers’ perceptions of the portal, its utility, and how it could be enhanced.
Study Overview and Design
The study was conducted by a team with diverse organizational roles, including clinical directors, a project manager, researchers, and a family-centered care specialist who is a parent of a client. The team brought different perspectives and backgrounds (social work, occupational therapy, and psychology) to the design of the study and interpretation of the findings.
Asshows, we used 3 methods of data collection: capture of portal login information, a survey (assessing utility and satisfaction, and impact on client engagement and perceptions of caregiver-provider communication), and focus groups or interviews held with caregivers and service providers. To ensure the opportunity for a base of experience prior to assessment, participants had a minimum of 2 months’ exposure to the EHR and 1.5 months to e-messaging before time 1 assessment. There was an interval of 6 to 8 weeks between time 1 and time 2 measurement points, allowing us to examine the effects of additional exposure to the portal. Caregivers could opt into 1 of 2 arms of the study: survey only; or survey plus focus group or interview. Service providers participated only in focus groups or interviews.
We obtained ethics approval from Holland Bloorview’s Research Ethics Board. Members of Holland Bloorview’s Family Advisory Committee introduced caregivers to the study when they were registering for the portal. Interested caregivers provided their contact information to learn more about the study from a research assistant. Other recruitment methods were (1) study flyers, (2) emails sent to connect2care users, (3) an advertisement posted on the Participate in Research page on the hospital’s website, and (4) messages posted on the hospital’s Facebook page for parents.
We administered study forms for caregivers (invitation, background form, and surveys) online using Research Electronic Data Capture (REDCap), a secure, Web-based app designed to support data capture for research studies ; all forms and materials were in English, as this is the language used in connect2care. Other than the ability to read English, there were no inclusion or exclusion criteria. Caregivers (of clients of any age or diagnosis) accessed a link on the portal’s home page that provided access to REDCap. The REDCap link contained a downloadable PDF version of the information letter and a consent form. At this time, caregivers indicated their interest in 1 of 2 study options (survey only or survey plus focus group), and provided their contact information so that study information and consent forms could be sent to them by email. As part of providing consent, caregivers gave permission for their portal usage information to be included in the research.
We recruited service providers using an announcement on the hospital’s internal home page, as well as emails with the study information flyer attached. Interested service providers were emailed the background form, which they returned to the research assistant.
The background form for caregivers captured demographic information (sex, education), and information about their child or adolescent with a disability (primary diagnosis). It also captured Internet use and ratings of Internet skills , since lack of Internet experience is the primary barrier to portal use [ ]. The background form for service providers captured discipline, education, and years in practice.
Since existing surveys did not capture the outcomes of interest, we developed our own. Satisfaction and utility items were informed by surveys developed to evaluate patient access to EHRs (eg, [, , ]). Engagement with care items were informed by the Pediatric Rehabilitation Intervention Measure of Engagement for Parents (GK, unpublished measure, 2015), and caregiver-provider communication items were based on constructs from existing surveys [ , ].
We refined the survey items, piloted them with 6 caregivers to ensure the items were acceptable and easily understood, and then reviewed them for health literacy. Caregivers indicated that the surveys took less than 10 minutes to complete. Based on caregivers’ feedback, we clarified the wording (eg, “complete” meant not missing important information), added definitions of key terms to the instruction section (eg, clinical provider team, care, and involved or engaged), and instructed respondents to click on “not applicable” or “to a small extent or not at all” when they had not had relevant opportunities to do what items specified.
The final survey (Portal Survey on Satisfaction and Impact on Care ) contained 38 items in 5 sections: (1) utility/satisfaction: client health record, (2) utility/satisfaction: portal messaging, (3) involvement in the care process, (4) impact of portal messaging with service providers (eg, improvements to communication, ability to express concerns and get clarifications, trust or rapport with clinical provider team), and (5) portal overall (ie, satisfaction and utility overall, future intention to use, impact on care).
Focus Groups and Interviews
Focus groups and interviews were conducted by an interviewer with more than 10 years of interview experience. Individual interviews were conducted (either by phone or in person) when individuals could not attend scheduled focus groups. Focus groups and interviews were audio-recorded and transcribed by a trained transcriptionist, with identifying information removed. Focus groups lasted an average of 48 minutes (range 35-60 minutes), and individual interviews lasted an average of 19 minutes (range 7-34 minutes). Participants were asked to discuss (1) general thoughts about the portal (including satisfaction and the extent to which they accessed the portal, and for what purposes), (2) how the portal could be enhanced or improved, (3) the most helpful or useful parts of the portal, (4) difficulties or issues in using or accessing the portal, (5) needs and expectations regarding the portal, and (6) whether the portal made things more efficient.
Portal Usage by Caregivers
Decision Support and Health Information Management at the hospital provided the study research assistant with an Excel 97-2003 (Microsoft Corporation) workbook of data covering the 14-month study period. We used this information to create a dataset containing the portal pages accessed by date for each person enrolled in the research. We then used this information to calculate total number of visits and days of use per person, usage across months per person, and the number of times each page was viewed over all participants.
We analyzed time 1 and time 2 survey data descriptively, given the small sample size. Aggregate scores were calculated for survey scales (the 5 survey sections), and Cronbach alpha was calculated to determine the scales’ internal consistency reliabilities.
We analyzed transcripts using a content analysis approach, which involves coding statements based on key concepts, combining these coded concepts into themes, and then refining the identified themes [, ]. Team members read all caregiver transcripts. To ensure confidentiality, service provider transcripts were analyzed by 3 team members (not clinical or project directors). Through an iterative process, the lead author then summarized the transcript data into themes, using tables to group and compare related ideas. The credibility and accuracy of data analysis were ensured by using multiple procedures, including maintaining an audit trail [ ]. Trustworthiness was enhanced through peer examination and discussion of findings in team meetings, investigator triangulation (various disciplines, perspectives, and roles), and mutual confirmation of the data [ , ].
Description of Participants
A total of 18 caregivers took part, some in the survey-only option and some in the survey plus focus group or interview. Asshows, some individuals participated at both time points. There were 15 completed surveys at time 1 and 11 at time 2.
presents the participants’ characteristics for the entire sample. As well, almost all participants (n=17, 94%) had used the Internet for more than 5 years, and all reported using it daily or several days a week. They rated their Internet skills as either good to very good (n=12, 67%) or average to reasonable (n=6, 33%).
|Age range (years)|
|Highest education level|
|High school certificate or diploma||1||6|
|College or other nonuniversity certificate or diploma||3||17|
|University certificate or diploma (eg, bachelor’s)||9||50|
|Postgraduate certificate or diploma (eg, master’s, PhD)||5||28|
|Child’s current primary health or development problem|
|Acquired brain injury||4||22|
|Spina bifida or hydrocephalus||2||11|
|Other (eg, global developmental delay, muscular dystrophy, central nervous system vasculitis or stroke)||5||28|
Service Provider Participants
Our target was to recruit 8 to 10 service providers for the focus groups, as is typical in focus group research . A total of 5 service providers took part in focus group or interviews: 2 physical therapists, 1 occupational therapist, 1 speech-language pathologist, and 1 nurse; 4 of the 5 participating service providers took part at both time points. They had been in practice 13.5 years on average; 2 had a bachelor’s and 3 had a master’s degree.
Portal Usage by Caregivers
presents portal usage information over the study period, from the introduction of the portal (January 2015) to the end of data collection (March 2016). As shows, we collected login information from first use of the portal, whereas we held surveys and focus groups or interviews only after participants had a minimum of 2 months’ exposure to the portal (to ensure a base of experience). Users joined connect2care at various points over this time period, with an average exposure of 253 days (approximately 9 months). Overall, users logged on to the portal an average of 22.2 times on 19.2 days (2.5 times a month). The most common user access pattern was a combination of home page, health record main page, appointment main page, and reports main page. Thus, users were most interested in their child’s health record, appointments, and reports, as has been reported by others [ ].
Login graphs for participants indicated differences in patterns of usage, with some users having high initial use that tapered off, whereas others had no use past their initial portal logon. The typical pattern, though, was a steady level of use (2 or 3 times a month). We chose the graphs into show typical login patterns over the study period.
|Number of days with exposure to connect2care (last session date minus first session date)||253||1-433|
|Number of times logged in to connect2care||22.2||1-87|
|Number of days logged in to connect2care||19.2||1-69|
Survey Results at Time 1 and Time 2
shows mean scores for the survey scales, along with the scales’ internal consistencies. Cronbach alpha values ranged from .89 to .95, indicating excellent internal consistency reliability [ , ]. There was a moderate extent of agreement with respect to utility of and satisfaction with the client health record and portal messaging. Scores for involvement in the care process and for the portal overall indicated agreement between “neutral” (score of 0) and “agree” (score of 1). Mean scores for portal messaging with service providers were neutral (–0.06) at time 1 and slightly higher (0.25) at time 2.
Mean Survey Item Responses
shows a fair degree of overlap in items with the highest and lowest means at time 1 and time 2. The items with the highest means (greatest endorsement) concerned the accuracy and convenience of the client health record (section A), the usefulness, timeliness, and adequacy of portal messaging (B), a willingness to invest effort in the intervention process (C), and convenience in accessing care or services (D). There was also evidence of a strong intention to use the portal in the future (E). Items with the lowest means (least endorsement) concerned actual use of the client health record (although this was moderate; section A) and portal messaging (to a fairly small extent; B). There was little evidence that the portal led to feelings of greater involvement in the care process (C), improved ability to express concerns to providers or enhanced relationships with providers (D), or reduced number of in-person visits (E).
|Survey section||No. of items||Cronbach alpha |
(times 1 and 2 combined)
|Time 1 (n=15)||Time 2 (n=11)|
|Utility/satisfaction: client health recordb||6||.89||3.68||0.76||3.14||1.01|
|Utility/satisfaction: portal messagingb||6||.90||2.62||1.08||3.22||1.11|
|Involvement in the care processc||10||.91||0.69||0.77||0.77||0.52|
|Impact of portal messaging with service providersc||7||.95||–0.06||0.67||0.25||0.99|
aSome data are missing (a score was calculated only if a respondent had data for ≥50% of the items in the scale).
bScale choices were to a great extent=5, to a fairly great extent=4, to a moderate extent=3, to a fairly small extent=2, to a small extent or not at all=1.
cScale choices were strongly agree=2, agree=1, neutral=0, disagree= –1, strongly disagree= –2.
|Survey section||Time 1 (n=15)a||Time 2 (n=11)a|
|Item(s) with the highest mean||Item(s) with the lowest mean||Item(s) with the highest mean||Item(s) with the lowest mean|
|A. Utility/satisfaction: client health recordb|
To what extent did you...
|Feel the client health record was accurate (there were no mistakes)? (mean 4.07)||Access your/your child’s health record? (mean 2.93)||Find it easy to locate relevant information in the client health record? (mean 3.82)||Access your/your child’s health record? (mean 2.45)|
|B. Utility/satisfaction: portal messagingb|
To what extent did you...
|Feel this feature was useful to you? (mean 3.60)||Communicate with a member of the provider team using portal messaging? (mean 1.70)||Feel responses to your portal messages were timely? (mean 3.83) |
Feel responses to your portal messages were adequate? (mean 3.83)
|Communicate with a member of the provider team using portal messaging? (mean 2.00)|
|C. Involvement in the care processc|
Based on my portal
|I am willing to invest effort in the intervention process (mean 1.20)||I believe the intervention process (ie, treatment plan) will be effective (due to greater involvement in the care process) (mean 0.40)||I am willing to invest effort in the intervention process (mean 1.27)||I believe the intervention process (ie, treatment plan) will be effective (due to greater involvement in the care process) (mean 0.18)|
|D. Impact of portal messaging with service providersc|
Because of the portal...
|I feel that accessing care or services is more convenient for me (mean 0.67)||My ability to express concerns and/or provide comments to providers has improved (mean –0.36) |
I have built a more open and trusting relationship with health care providers (mean –0.36)
|I feel that accessing care or services is more convenient for me (mean 0.64)||My communication with my provider has improved (mean 0.10) |
I have built a more open and trusting relationship with health care providers (mean 0.10)
My sense of trust/rapport with the clinical provider team has increased (mean 0.10)
|E. Portal overallc||I intend to use the portal in the future (mean 1.47)||The portal reduced the number of in-person visits I made (mean –0.27)||I intend to use the portal in the future (mean 1.00)||The portal reduced the number of in-person visits I made (mean –0.36)|
aSome data are missing (a score was calculated only if a respondent had data for ≥50% of the items in the scale).
bScale choices were to a great extent=5, to a fairly great extent=4, to a moderate extent=3, to a fairly small extent=2, to a small extent or not at all=1.
cScale choices were strongly agree=2, agree=1, neutral=0, disagree= –1, strongly disagree= –2.
Three themes arose from our analysis of the caregiver transcripts, with the nature of these changing slightly at time 2 as caregivers acquired more experience with the portal. These themes were information benefits, recommendations to increase use and utility, and scope of adoption and future vision.
With respect to information benefits, at time 1, caregivers indicated the usefulness of the portal in providing easy access to their child’s medical history, reports, and appointments, and how this saved them time. At time 2, the benefits of access to the information itself received somewhat greater emphasis—participants expressed an appreciation for having more detailed information and knowing the technical language, as they felt they could then communicate on a more level playing field with providers (eg, “[if] we can use accurate language, language that they understand, we get a lot better dynamic where they’re going to listen to us”).
There were many recommendations to increase use and utility, including being able to message all clinicians, receiving speedier notification of messages waiting for them in the portal (caregivers had to go into connect2care to get their emails, and notifications about awaiting emails took time to arrive) and better information about how to do things on the portal (eg, send a message), and having clarification of the scope of confidentiality and portal access after discharge. The recommendations also included uniform implementation of portal features (some caregivers did not have access to features that others did) and greater comprehensiveness in what was available (eg, medication information, care plans, and particularly reports from all clinicians). These comments may reflect differences in whether a caregiver’s service providers were set up on e-messaging.
The theme of scope of adoption and future vision concerned caregivers’ ultimate hopes for the portal. They were aware that the portal was a “work in progress” and “in its infancy,” and shared their visions of the portal in the future. They commented on the utility of cross-organization EHRs, connection to the adult health care system, provision of information about available programs, and personalization on a broad scale (eg, ability to upload one’s own information to the portal and receive targeted information). They also commented on the need for organizationwide adoption, where portal use was embedded in routine practice (eg, “it’s early days...but if just everybody could get on board...the opportunities for saving everybody’s time and aggravation in messaging are huge”).
Service Provider Themes
Four themes emerged from our analysis of the service provider transcripts. There were no differences in these themes at time 1 and time 2. First, the theme utility for families indicated that service providers saw the utility of the portal in setting up appointments and providing secure messaging. They were less sure about the impact of the portal on client engagement but felt that the portal provided a positive, inviting message to families about being engaged (eg, “messages of we want you to be engaged, we’re trying to reach out to you, we’re trying to have multiple ways of engaging with you”).
Second, service providers identified technical shortcomings in several areas, including lack of notification of emails (requiring repeated logons to check whether emails from caregivers had been received), and lack of ability to post vacation messages and upload attachments. Problems in formatting occurred when reports from the clinical system were uploaded to connect2care, and reports had to be changed in several places to ensure consistency in what clients saw.
The third theme dealt with uncertainties in portal use, related to lack of knowledge, comfort, or confidence in using the portal, in addition to some of the portal’s technical shortcomings. Service providers expressed uncertainty in knowing which families accessed the portal, whether reports were being used, whether a message was waiting or had been received, and what documents or materials families could access. They were also uncertain about their role in informing families about the portal and how to access it (eg, “it doesn’t seem like a big deal, but when I’m seeing 5, sometimes 6 people in a day, I don’t have that extra time to do that”).
The fourth service provider theme concerned use, effort, and investment in the portal. Due to low levels of perceived use by families, and the time, effort, and care required to see if messages were waiting and to produce user-friendly reports meeting professional standards, service providers expressed concerns about whether it was currently worth investing a great amount of time in the portal (eg, “it’s another thing to log in to,” “doesn’t make sense for me to go in every day, or multiple times a day, and check if in 6 months I’ve only ever got 2 messages”).
This study contributes to the growing literature on portals, as there has been no previous research, to our knowledge, on the use, utility, or impact of client portals in pediatric rehabilitation. Furthermore, studies of health care portals have largely focused on use and utility, rather than impact-related outcomes such as engagement in care or client-provider communication . Compared with new portal annual adoption rates of 5% to 10% and access rates of 0.4 to 0.6 uses per month per user [ ], the connect2care portal had an adoption rate of 12.4% and 2.5 logins a month per user over 9 months of exposure. There may be many reasons for the higher adoption and access rates, including the involvement of families in the development of the portal [ ], effort put into informing families about the portal, and differences between acute care and pediatric rehabilitation contexts.
There was a moderate degree of perceived usefulness of and satisfaction with the EHR and e-messaging features, and evidence that the portal was perceived to provide useful access to the clinical record. Utility and satisfaction scores did not change much over time; however, there was only a 6- to 8-week time period between the 2 survey administrations, thus reducing the amount of new portal experience that was possible.
With respect to impact, there was some evidence that portal access facilitated caregivers’ perceptions of engagement in care, but this evidence was not strong (between neutral and agree, on average). As a point of comparison, van der Vaart et al  found that 44% of patients with rheumatoid arthritis felt more involved in their treatment as a result of portal access. Service providers in our study also indicated seeing little evidence of increased engagement. However, several caregivers indicated that reading available reports increased their understanding of the related technical language, and felt this enhanced their ability to engage in conversations about care with service providers.
We speculate that it takes time for portal use to have an effect on engagement in care (in this study, we collected time 1 engagement data after an average of approximately 9 months of portal exposure). At this early stage of portal use, caregivers appeared predominantly interested in the information they could access via the portal, although at the second measurement point 2 months later, there was some indication of greater appreciation of the actual content of the reports, which may reflect the fact that more and more clinical documentation was being added through the study period. With respect to impact on caregiver-provider communication, there was some evidence at time 2 of improved communication due to the introduction of e-messaging. We speculate that e-messaging had not been introduced long enough or widely enough to affect client engagement. Alternatively, caregivers may be already quite engaged in their child’s care, so that the portal may not make such a big difference in engagement compared with nonpediatric portals.
The qualitative themes were informative, as they allowed comparison of caregivers’ and service providers’ perspectives. There was a common emphasis on the utility of the information provided via the portal. As well, both groups questioned the extent of the impact on engagement in care at this early stage of portal rollout (corroborating survey data), and both groups recommended increasing portal use and utility by addressing technical shortcomings. These recommendations were related to the portal not meeting expectations for technology and not being as user friendly as desired.
The major difference between caregivers and providers was that caregivers focused on the scope of adoption of the portal system in the organization and expressed their hopes for the future of the portal with respect to their family’s life journey (the scope of adoption and future vision theme), whereas service providers were concerned about how to best manage their investment of time and effort (the use, effort, and investment in the portal theme). Both groups expressed a desire for the other group to use the portal to a greater extent: caregivers wanted to see organizationwide adoption, whereas some providers in our small sample questioned whether their investment in the portal was justified given low levels of perceived use. As well as full organizationwide adoption of the portal system, caregivers’ hopes for the future included greater personalization and comprehensiveness of the provided information. As recognized by caregivers, portals are in their infancy and the maturity of portals does not appear to be where it needs to be to improve quality of care and involve the patient in care decisions .
Planned future enhancements to the connect2care portal will address some of the hopes of caregivers, including the ability to import reports from other service providers, link to records from other hospitals, and receive materials and resources specifically targeted to their care needs .
Study Strengths and Limitations
Strengths of this investigation include prospective data collection of login information starting at the portal’s introduction, the breadth of information collected using multiple methods, inclusion of both caregiver and provider perspectives, and focused examination of caregiver engagement and caregiver-provider communication using scales with excellent internal consistency. Study limitations include its descriptive nature, the relatively short time period between measurement points (6-8 weeks), and the relatively small number of respondents (18 caregivers and 5 providers). We likely did not have data saturation and therefore the robustness of the qualitative themes is uncertain [, ], although triangulation with the survey data should be noted.
The study participants are likely representative of those who register for portal access but not of the Holland Bloorview client population overall, since people who register for portals are likely to be early adopters who embrace technology more readily. Research suggests that those who use patient portals are generally more highly educated, younger, more affluent, and have fewer medical problems than nonenrollees [, , ]. As with any innovation, expectations and technical difficulties play a role in adoption. Difficulties in getting portal technology to run as desired can affect portal use and may bias reports of utility, satisfaction, and impact [ , ], as may be the case in our study.
Research and Organizational Implications
Future research directions include continued use of the survey to see whether there is evidence of increasing impact of the portal on client outcomes. Other directions include examination of expectations that are held about portal use and utility, and examination of the perceptions of young adult clients. For other researchers, this study has indicated the utility of analyzing login statistics and using a portal survey with demonstrated internal consistency. The study has also indicated the utility of an integrated knowledge translation approach, where clinicians, family members, and researchers come together to address an important applied question .
Organizationally, the findings indicate specific areas for improvements to the portal and its processes, many of which are being or have been addressed as the portal continues to be developed, including the establishment of processes to improve efficiency related to portal activities (such as reminder emails when a new message is received) and mechanisms to support the enrollment of all interested clients and caregivers. The findings also suggest the need for ongoing education about portal intents and for transparency in communication about implementation issues. The literature indicates that education is needed to manage expectations and enhance the extent of portal adoption . Education and resources are needed to support providers in feeling confident that their clinical documentation is family-friendly and in using e-messaging effectively. Similarly, education is needed to enable caregivers to make the best possible use of the portal features.
Although caregivers in our sample saw the portal’s value, this was not yet the case for the service providers. This suggests that organizations may need to focus some of their efforts on ensuring that service providers see this value, as they are the ones who will share clinical notes and communicate via e-messaging—they too are partners in care. In directing attention to engaging clients or families, the engagement of service providers can be given less attention. As technology becomes more prominent in health care settings, it will be up to organizations to support uptake and demonstrate potential benefits for both parties. An important future direction at Holland Bloorview is therefore to share family stories with providers so that they can better understand the portal’s positive impact and feel that their time related to sharing information and communicating via the portal is worth the effort.
Lastly, the findings endorse the often-made statement that changing organizational culture takes time. Portal adoption is a process—not a one-time event—and requires a feedback loop (as provided by this study), allowing an organization to improve portal adoption through attention to the needs of the people who use it.
Canada Health Infoway Inc. funded the demonstration project supporting the development and evaluation of Holland Bloorview Kids Rehabilitation Hospital’s client/family EHR portal. Canada Health Infoway Inc. also covered the publication costs for this article. Canada Health Infoway Inc. is an independent, not-for-profit organization funded by the federal government that works to accelerate the development, adoption, and effective use of digital health across Canada. The funders had no role in the data collection or decision to publish the manuscript. Simon Hagens (affiliated with Canada Health Infoway) was a member of the research team and involved in study design and interpretation. Gillian King holds the Canada Research Chair in Optimal Care for Children with Disabilities, funded by the Canadian Institutes of Health Research.
We thank Siamak Tenzif for assistance with obtaining information about usage patterns and frequency of login information, Becky Quinlan for assistance with registering people for the portal, and Alanna Rudzik for conducting the focus groups and interviews. We would also like to thank those who participated in the study and members of the Family Advisory Committee who provided input on the survey.
Conflicts of Interest
Simon Hagens is affiliated with Canada Health Infoway.
- Kaelber DC, Jha AK, Johnston D, Middleton B, Bates DW. A research agenda for personal health records (PHRs). J Am Med Inform Assoc 2008;15(6):729-736 [FREE Full text] [CrossRef] [Medline]
- Maxwell J, Williams L, Adamson K, Karmali A, Quinlan B. Supporting client and family engagement in care through the planning and implementation of an online consumer health portal. J Patient Experience 2016 Aug 11;3(2):48-51. [CrossRef]
- Rosenbaum P, King S, Law M, King G, Evans J. Family-centred service: a conceptual framework and research review. Phys Occup Ther Pediatr 2009 Jul 29;18(1):1-20. [CrossRef]
- King G, Currie M, Petersen P. Child and parent engagement in the mental health intervention process: a motivational framework. Child Adolesc Ment Health 2012 Dec 14;19(1):2-8. [CrossRef]
- Davidson J, Wiens S, Anderson K. Creating a provincial family council to engage youth and families in child & youth mental health systems. J Can Acad Child Adolesc Psychiatry 2010 Aug;19(3):169-175 [FREE Full text] [Medline]
- Tetley A, Jinks M, Huband N, Howells K. A systematic review of measures of therapeutic engagement in psychosocial and psychological treatment. J Clin Psychol 2011 Sep;67(9):927-941. [CrossRef] [Medline]
- King G, Cathers T, King S, Rosenbaum P. Major elements of parents’ satisfaction and dissatisfaction with pediatric rehabilitation services. Child Health Care 2010 Jun 07;30(2):111-134. [CrossRef]
- Duffy FD, Gordon GH, Whelan G, Cole-Kelly K, Frankel R, Buffone N, Participants in the American Academy on Physician and Patient's Conference on Education and Evaluation of Competence in Communication and Interpersonal Skills. Assessing competence in communication and interpersonal skills: the Kalamazoo II report. Acad Med 2004 Jun;79(6):495-507. [Medline]
- Mercer LM, Tanabe P, Pang PS, Gisondi MA, Courtney DM, Engel KG, et al. Patient perspectives on communication with the medical team: pilot study using the Communication Assessment Tool-Team (CAT-T). Patient Educ Couns 2008 Nov;73(2):220-223. [CrossRef] [Medline]
- King G, Desmarais C, Lindsay S, Piérart G, Tétreault S. The roles of effective communication and client engagement in delivering culturally sensitive care to immigrant parents of children with disabilities. Disabil Rehabil 2015;37(15):1372-1381. [CrossRef] [Medline]
- Stylus Consulting. Nova Scotia Personal Health Record demonstration project: benefits evaluation report. Toronto, ON: Canada Health Infoway; 2014. URL: https://www.infoway-inforoute.ca/en/component/edocman/1995-nova-scotia-personal-health-record-demonstration-project-benefits-evaluation-report/view-document [accessed 2017-03-27] [WebCite Cache]
- Tang PC, Ash JS, Bates DW, Overhage JM, Sands DZ. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006;13(2):121-126 [FREE Full text] [CrossRef] [Medline]
- Hassol A, Walker JM, Kidder D, Rokita K, Young D, Pierdon S, et al. Patient experiences and attitudes about access to a patient electronic health care record and linked web messaging. J Am Med Inform Assoc 2004;11(6):505-513 [FREE Full text] [CrossRef] [Medline]
- van der Vaart R, Drossaert CH, Taal E, Drossaers-Bakker KW, Vonkeman HE, van de Laar MA. Impact of patient-accessible electronic medical records in rheumatology: use, satisfaction and effects on empowerment among patients. BMC Musculoskelet Disord 2014;15:102 [FREE Full text] [CrossRef] [Medline]
- Ahmad FS, Tsang T. Diabetes prevention, health information technology, and meaningful use: challenges and opportunities. Am J Prev Med 2013 Apr;44(4 Suppl 4):S357-S363. [CrossRef] [Medline]
- Patterson ES, Zhang J, Abbott P, Gibbons MC, Lowry SZ, Quinn MT, et al. Enhancing electronic health record usability in pediatric patient care: a scenario-based approach. Jt Comm J Qual Patient Saf 2013 Mar;39(3):129-135. [Medline]
- Power TJ, Michel J, Mayne S, Miller J, Blum N, Grundmeier R. Coordinating systems of care using health information technology: development of the ADHD Care Assistant. Adv School Ment Health Promotion 2016;9(3-4):201-218.
- Fiks AG, DuRivage N, Mayne SL, Finch S, Ross ME, Giacomini K, et al. Adoption of a portal for the primary care management of pediatric asthma: a mixed-methods implementation study. J Med Internet Res 2016 Jun 29;18(6):e172 [FREE Full text] [CrossRef] [Medline]
- Kelly MM, Hoonakker PL, Dean SM. Using an inpatient portal to engage families in pediatric hospital care. J Am Med Inform Assoc 2017 Jan;24(1):153-161. [CrossRef] [Medline]
- Children's Hospital of Eastern Ontario. Benefits evaluation report: Canada Health Infoway deliverable AMBHIS-17B. Toronto, ON: Canada Health Infoway; 2016 Feb 29. URL: https://www.infoway-inforoute.ca/en/component/edocman/3032-children-s-hospital-for-eastern-ontario-epiccare-ambulatory-program-benefits-evaluation-report/view-document [accessed 2017-03-27] [WebCite Cache]
- Tenforde M, Jain A, Hickner J. The value of personal health records for chronic disease management: what do we know? Fam Med 2011 May;43(5):351-354 [FREE Full text] [Medline]
- Chen C, Garrido T, Chock D, Okawa G, Liang L. The Kaiser Permanente Electronic Health Record: transforming and streamlining modalities of care. Health Aff (Millwood) 2009 Apr;28(2):323-333 [FREE Full text] [CrossRef] [Medline]
- Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. JAMA 2012 Nov 21;308(19):2012-2019. [CrossRef] [Medline]
- Emont S. Measuring the impact of patient portals: what the literature tells us. Oakland, CA: California HealthCare Foundation; 2011 May. URL: http://www.chcf.org/~/media/MEDIA%20LIBRARY%20Files/PDF/PDF%20M/PDF%20MeasuringImpactPatientPortals.pdf [accessed 2017-03-27] [WebCite Cache]
- Urowitz S, Smith K, Alkazaz N, Apatu E, Quartey NK, Wiljer D. Patient accessible electronic health records for the chronically ill: a review of the literature. J Hosp Admin 2012 Nov 01;1(2):64-72. [CrossRef]
- Ammenwerth E, Schnell-Inderst P, Hoerbst A. The impact of electronic patient portals on patient care: a systematic review of controlled trials. J Med Internet Res 2012;14(6):e162 [FREE Full text] [CrossRef] [Medline]
- Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015;17(2):e44 [FREE Full text] [CrossRef] [Medline]
- Weingart SN, Rind D, Tofias Z, Sands DZ. Who uses the patient internet portal? The PatientSite experience. J Am Med Inform Assoc 2006;13(1):91-95 [FREE Full text] [CrossRef] [Medline]
- Currie M, King G, Rosenbaum P, Law M, Kertoy M, Specht J. A model of impacts of research partnerships in health and social services. Eval Program Plann 2005 Nov;28(4):400-412. [CrossRef]
- Hanson WE, Creswell JW, Clark VLP, Petska KS, Creswell JD. Mixed methods research designs in counseling psychology. J Couns Psychol 2005;52(2):224-235. [CrossRef]
- Hall B, Howard K. A synergistic approach: conducting mixed methods research with typological and systematic design considerations. J Mixed Methods Res 2008 Jul 01;2(3):248-269. [CrossRef]
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381 [FREE Full text] [CrossRef] [Medline]
- Nazi KM, Hogan TP, McInnes DK, Woods SS, Graham G. Evaluating patient access to electronic health records: results from a survey of veterans. Med Care 2013 Mar;51(3 Suppl 1):S52-S56. [CrossRef] [Medline]
- System and use survey: e-visits. Toronto, ON: Canada Health Infoway; 2015 Jun 11. URL: https://www.infoway-inforoute.ca/en/resource-centre/advanced-search?q=System+and+Use+Survey+-+e [accessed 2017-03-25] [WebCite Cache]
- Kristjansson E, Tugwell PS, Wilson AJ, Brooks PM, Driedger SM, Gallois C, et al. Development of the effective musculoskeletal consumer scale. J Rheumatol 2007 Jun;34(6):1392-1400. [Medline]
- Holland Bloorview Kids Rehabilitation Hospital. Portal Survey on Satisfaction and Impact on Care. flintbox. Chicago, IL: Wellspring Worldwide, LLC; 2017 Jan 04. URL: https://flintbox.com/public/project/30920 [accessed 2017-03-21] [WebCite Cache]
- Fiese BH, Bickham NL. Qualitative inquiry: an overview for pediatric psychology. J Pediatr Psychol 1998;23(2):79-86. [Medline]
- Lincoln Y, Guba E. Naturalistic Inquiry. Newbury Park, CA: Sage Publications; 1985.
- Creswell J. Qualitative Inquiry and Research Design. 2nd edition. Thousand Oaks, CA: Sage Publications; 2007.
- Krefting L. Rigor in qualitative research: the assessment of trustworthiness. Am J Occup Ther 1991 Mar;45(3):214-222. [Medline]
- Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today 2004 Feb;24(2):105-112. [CrossRef] [Medline]
- Krueger R, editor. Focus Groups: A practical Guide for Applied Research. 3rd edition. Thousand Oaks, CA: Sage Publications; 2000.
- Streiner D, Norman G. Health Measurement Scales: A Practical Guide to Their Development and Use. Oxford, UK: Oxford University Press; 1989:79-96.
- Tabachnik B, Fidell L. Using Multivariate Statistics. New York, NY: Harper Collins; 1989.
- Gheorghiu B, Hagens S. Use and maturity of electronic patient portals. Stud Health Technol Inform 2017;234:136-141. [Medline]
- Strauss A, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 2nd edition. Thousand Oaks, CA: Sage Publications; 1998.
- Leveille SG, Huang A, Tsai SB, Allen M, Weingart SN, Iezzoni LI. Health coaching via an internet portal for primary care patients with chronic conditions: a randomized controlled trial. Med Care 2009 Jan;47(1):41-47. [CrossRef] [Medline]
- Or CK, Karsh B. A systematic review of patient acceptance of consumer health information technology. J Am Med Inform Assoc 2009 Aug;16(4):550-560 [FREE Full text] [CrossRef] [Medline]
- Krause MS. Behavioral indexes of motivation for treatment. J Couns Psychol 1967;14(5):426-435. [CrossRef]
- King G, Thomson N, Rothstein M, Kingsnorth S, Parker K. Integrating research, clinical care, and education in academic health science centers. J Health Organ Manag 2016 Oct 10;30(7):1140-1160. [CrossRef] [Medline]
|EHR: electronic health record|
|REDCap: Research Electronic Data Capture|
Edited by G Eysenbach; submitted 14.10.16; peer-reviewed by S Lindsay, MC Trudel, K Nazi; comments to author 28.01.17; revised version received 22.02.17; accepted 15.03.17; published 06.04.17Copyright
©Gillian King, Joanne Maxwell, Amir Karmali, Simon Hagens, Madhu Pinto, Laura Williams, Keith Adamson. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 06.04.2017.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.