Consumers are asking for the ability to view their appointment schedules online, have electronic access to documentation and other resources, and have their questions answered over the Internet. E-scheduling, e-visits, and e-messaging are seen as important portal functions [11], as they allow information exchange and can potentially enhance consumers’ ability to manage their own health and health care [1]. By having access to a portal that does more than just provide information and data, consumers can become more active participants in their own health care [12]. Thus, portals have the potential to provide more personalized, collaborative, and effective care. There is, however, limited (but growing) research information on the use of electronic communication in health care, including electronic health records (EHRs) [1,13].

Research on EHRs has typically focused on adult medical care and management of chronic diseases such as rheumatoid arthritis [14] and diabetes [15] rather than children’s chronic conditions [16]. We searched for articles on portals in pediatric rehabilitation hospitals (using “pediatric” or “paediatric” with “online” and “portal,” with searches limited to 2015 and more recent). We found articles on portals for children with mental health conditions [17], with asthma [18], and those receiving acute care in a pediatric hospital [19]. We found no articles on portals in pediatric rehabilitation hospitals. A recent evaluation of a portal at a Canadian children’s acute care hospital indicated that, at 3 months after EHR implementation, service providers perceived a negative impact on their efficiency, productivity, and the quality of care they provided, whereas patients and families did not perceive a negative impact on care [20].

Even for adult chronic disease management, there is sparse evidence to support the ability of personal health records and EHRs to improve efficiencies, reduce costs, and improve outcomes [15,21]. Most existing research on health care portals has examined the frequency of use of their features, satisfaction with the portal, or efficiencies in health care utilization (eg, number of office and emergency department visits, phone calls) [22-24].

Systematic reviews on the effects of portals with EHR access [25-27] indicate that few studies have investigated outcomes concerning patient engagement or empowerment, with studies typically providing nonsignificant findings. Kruse et al [27] found only 27 studies of relevance to impact—those measuring meaningful outcomes such as patient participation in medical decisions, patient-provider communication, or satisfaction. For example, a study by van der Vaart and colleagues [14] found that approximately 44% of patients with rheumatoid arthritis reported feeling more involved in treatment as a result of access to their EHRs; however, significant pre-post differences in empowerment were not found. Thus, research is needed to substantiate “meaningful use” impacts [1,15].

To our knowledge, there has been no research on client portals for pediatric rehabilitation service users. As well, the literature has focused on portal use, satisfaction, and service utilization outcomes, and there is a recognized need to examine meaningful outcomes such as client engagement in care and communication with providers. These 2 outcomes are of fundamental importance in pediatric rehabilitation, as well as in medical services for people with chronic conditions [27]. Furthermore, although some studies have tracked portal use over time (eg, [28]), studies have not examined portal impact prospectively. Examining portal use, utility, and impact over time provides an opportunity to see how users access portal features and may also indicate emergent short-term impacts [29], such as enhanced engagement in care. Since studies typically evaluate portal features in isolation [27], there is benefit in examining client perceptions of the introduction of a full portal over a period of time. Lastly, the literature indicates the value of mixed-methods studies, which are rare in portal research [27].

Holland Bloorview Kids Rehabilitation Hospital (Holland Bloorview) in Toronto, Ontario, is Canada’s largest children’s rehabilitation hospital. The hospital is a provincial resource for children with cerebral palsy, acquired brain injury, muscular dystrophy, amputation, epilepsy, spina bifida, arthritis, cleft lip and palate, autism, and other physical and developmental disabilities. Holland Bloorview’s vision is to create a world of possibility for kids with disability by embracing client- and family-centered care, and participating in applied research and education.

In early 2015, Holland Bloorview launched a consumer portal with the ultimate goal of helping clients and their caregivers (family members, most typically parents) take an active role in managing their own or their children’s care. The consumer health portal (called connect2care) was developed in partnership with clients and families [2]. Connect2care provides clients and families with electronic access to their medical records, online appointment cancelling and booking features, transparent and timely access to clinical documentation, and e-messaging to connect with their care providers.

Beginning in January 2015, enrollment to the portal was initiated, first for clients on the inpatient units, and then also for ambulatory and community programs. In this first phase, the functions available included the ability to view the client’s schedule and visit history, view and print clinical notes, and update demographic details. In May 2015, improvements were made to the viewing of laboratory and microbiology test results, and new processes were established to increase the number of clinical notes that would flow to the portal (allowing providers from additional health disciplines to share clinical notes). Over the summer of 2015, training of more than 100 health care providers was completed to support this enhanced sharing of clinical notes. In the summer and fall of 2015, messaging functionality was rolled out, allowing secure e-messaging between portal users and their providers. By December 2015, the training of providers for e-messaging was completed.

Enrollment and usage targets for 2015 were established prior to launching the portal, with a target enrollment of 721 users (clients and caregivers), and a target use (unique logins) of 1440. The targets were met well ahead of schedule and, by the end of 2015, there were 869 enrolled users and more than 4800 uses. The adoption rate was approximately 12.41% (869 out of about 7000 unique clients). As well, over 200 staff were live with e-messaging at the end of 2015, including physicians, occupational therapists, physical therapists, speech-language pathologists, social workers, psychologists, therapeutic recreation staff, orthotics and prosthetics staff, ambulatory care nurses, child life workers, and nurse practitioners.

The purpose of our study was to examine the use, utility, and impact of the connect2care portal from the beginning of portal introduction until the end of data collection 14 months later. To meet this aim, we conducted a prospective, mixed-methods study collecting quantitative survey data and qualitative data from focus groups and interviews with caregivers and service providers at 2 points in time. We adopted a concurrent triangulation approach, in which quantitative and qualitative data were collected at the same time [30] and integrated at the level of interpretation [31]. Our research objectives were (1) to determine caregivers’ portal use over the study period, (2) to examine levels of perceived usefulness of and satisfaction with the portal’s EHR and e-messaging, as well as the portal’s impact on engagement in care and caregiver-provider communication, and (3) to ascertain caregivers’ and service providers’ perceptions of the portal, its utility, and how it could be enhanced.

The study was conducted by a team with diverse organizational roles, including clinical directors, a project manager, researchers, and a family-centered care specialist who is a parent of a client. The team brought different perspectives and backgrounds (social work, occupational therapy, and psychology) to the design of the study and interpretation of the findings.

As Figure 1 shows, we used 3 methods of data collection: capture of portal login information, a survey (assessing utility and satisfaction, and impact on client engagement and perceptions of caregiver-provider communication), and focus groups or interviews held with caregivers and service providers. To ensure the opportunity for a base of experience prior to assessment, participants had a minimum of 2 months’ exposure to the EHR and 1.5 months to e-messaging before time 1 assessment. There was an interval of 6 to 8 weeks between time 1 and time 2 measurement points, allowing us to examine the effects of additional exposure to the portal. Caregivers could opt into 1 of 2 arms of the study: survey only; or survey plus focus group or interview. Service providers participated only in focus groups or interviews.

The background form for caregivers captured demographic information (sex, education), and information about their child or adolescent with a disability (primary diagnosis). It also captured Internet use and ratings of Internet skills [14], since lack of Internet experience is the primary barrier to portal use [27]. The background form for service providers captured discipline, education, and years in practice.

Since existing surveys did not capture the outcomes of interest, we developed our own. Satisfaction and utility items were informed by surveys developed to evaluate patient access to EHRs (eg, [14,33,34]). Engagement with care items were informed by the Pediatric Rehabilitation Intervention Measure of Engagement for Parents (GK, unpublished measure, 2015), and caregiver-provider communication items were based on constructs from existing surveys [33,35].

We refined the survey items, piloted them with 6 caregivers to ensure the items were acceptable and easily understood, and then reviewed them for health literacy. Caregivers indicated that the surveys took less than 10 minutes to complete. Based on caregivers’ feedback, we clarified the wording (eg, “complete” meant not missing important information), added definitions of key terms to the instruction section (eg, clinical provider team, care, and involved or engaged), and instructed respondents to click on “not applicable” or “to a small extent or not at all” when they had not had relevant opportunities to do what items specified.

The final survey (Portal Survey on Satisfaction and Impact on Care [36]) contained 38 items in 5 sections: (1) utility/satisfaction: client health record, (2) utility/satisfaction: portal messaging, (3) involvement in the care process, (4) impact of portal messaging with service providers (eg, improvements to communication, ability to express concerns and get clarifications, trust or rapport with clinical provider team), and (5) portal overall (ie, satisfaction and utility overall, future intention to use, impact on care).

Focus groups and interviews were conducted by an interviewer with more than 10 years of interview experience. Individual interviews were conducted (either by phone or in person) when individuals could not attend scheduled focus groups. Focus groups and interviews were audio-recorded and transcribed by a trained transcriptionist, with identifying information removed. Focus groups lasted an average of 48 minutes (range 35-60 minutes), and individual interviews lasted an average of 19 minutes (range 7-34 minutes). Participants were asked to discuss (1) general thoughts about the portal (including satisfaction and the extent to which they accessed the portal, and for what purposes), (2) how the portal could be enhanced or improved, (3) the most helpful or useful parts of the portal, (4) difficulties or issues in using or accessing the portal, (5) needs and expectations regarding the portal, and (6) whether the portal made things more efficient.

Table 2 presents portal usage information over the study period, from the introduction of the portal (January 2015) to the end of data collection (March 2016). As Figure 1 shows, we collected login information from first use of the portal, whereas we held surveys and focus groups or interviews only after participants had a minimum of 2 months’ exposure to the portal (to ensure a base of experience). Users joined connect2care at various points over this time period, with an average exposure of 253 days (approximately 9 months). Overall, users logged on to the portal an average of 22.2 times on 19.2 days (2.5 times a month). The most common user access pattern was a combination of home page, health record main page, appointment main page, and reports main page. Thus, users were most interested in their child’s health record, appointments, and reports, as has been reported by others [20].

Login graphs for participants indicated differences in patterns of usage, with some users having high initial use that tapered off, whereas others had no use past their initial portal logon. The typical pattern, though, was a steady level of use (2 or 3 times a month). We chose the graphs in Figure 3 to show typical login patterns over the study period.

Strengths of this investigation include prospective data collection of login information starting at the portal’s introduction, the breadth of information collected using multiple methods, inclusion of both caregiver and provider perspectives, and focused examination of caregiver engagement and caregiver-provider communication using scales with excellent internal consistency. Study limitations include its descriptive nature, the relatively short time period between measurement points (6-8 weeks), and the relatively small number of respondents (18 caregivers and 5 providers). We likely did not have data saturation and therefore the robustness of the qualitative themes is uncertain [38,46], although triangulation with the survey data should be noted.

The study participants are likely representative of those who register for portal access but not of the Holland Bloorview client population overall, since people who register for portals are likely to be early adopters who embrace technology more readily. Research suggests that those who use patient portals are generally more highly educated, younger, more affluent, and have fewer medical problems than nonenrollees [28,47,48]. As with any innovation, expectations and technical difficulties play a role in adoption. Difficulties in getting portal technology to run as desired can affect portal use and may bias reports of utility, satisfaction, and impact [33,49], as may be the case in our study.

Future research directions include continued use of the survey to see whether there is evidence of increasing impact of the portal on client outcomes. Other directions include examination of expectations that are held about portal use and utility, and examination of the perceptions of young adult clients. For other researchers, this study has indicated the utility of analyzing login statistics and using a portal survey with demonstrated internal consistency. The study has also indicated the utility of an integrated knowledge translation approach, where clinicians, family members, and researchers come together to address an important applied question [50].

Organizationally, the findings indicate specific areas for improvements to the portal and its processes, many of which are being or have been addressed as the portal continues to be developed, including the establishment of processes to improve efficiency related to portal activities (such as reminder emails when a new message is received) and mechanisms to support the enrollment of all interested clients and caregivers. The findings also suggest the need for ongoing education about portal intents and for transparency in communication about implementation issues. The literature indicates that education is needed to manage expectations and enhance the extent of portal adoption [27]. Education and resources are needed to support providers in feeling confident that their clinical documentation is family-friendly and in using e-messaging effectively. Similarly, education is needed to enable caregivers to make the best possible use of the portal features.

Although caregivers in our sample saw the portal’s value, this was not yet the case for the service providers. This suggests that organizations may need to focus some of their efforts on ensuring that service providers see this value, as they are the ones who will share clinical notes and communicate via e-messaging—they too are partners in care. In directing attention to engaging clients or families, the engagement of service providers can be given less attention. As technology becomes more prominent in health care settings, it will be up to organizations to support uptake and demonstrate potential benefits for both parties. An important future direction at Holland Bloorview is therefore to share family stories with providers so that they can better understand the portal’s positive impact and feel that their time related to sharing information and communicating via the portal is worth the effort.

Lastly, the findings endorse the often-made statement that changing organizational culture takes time. Portal adoption is a process—not a one-time event—and requires a feedback loop (as provided by this study), allowing an organization to improve portal adoption through attention to the needs of the people who use it.