For this study, we employed surveys and system audit logs. This combination of longitudinal semistructured user surveys and administrative data audit was used to identify user-reported outcomes triangulated with possible clinical outcomes of interest in a cohort of users attending clinics allowing access to a patient portal. The University of Alberta’s Health Research Ethics Board (Pro00066287) approved the study.

The patient portal that we studied is tethered to a shared EMR known as eCLINICIAN (an AHS branding of EpicCare Ambulatory 2014 from Epic Systems), used at the time by approximately 1110 physicians, residents, and students, and approximately 6000 nurses, allied health professionals, and other staff. The adaptation of this patient portal to the AHS context took about six months, the bulk of which was spent achieving consensus (including patient, clinical, administrative, legislative, and regulatory college input) on policy decisions and functions (Textbox 1). Consensus was reached to release most high-value data in real time (ie, those data required to manage active or chronic disease), with higher stakes information (ie, results of hepatitis or biopsies) delayed by ten days (Textbox 2).

AHS is a provincial health system responsible for the delivery of inpatient and ambulatory care services. Edmonton is the capital city of Alberta, with a population of approximately one million. In January 2016, an EMR-tethered patient portal was introduced in the Edmonton zone, and was evaluated between January 2016 and July 2018. During this phase, the patient portal was introduced in one family practice clinic and four specialty clinics (rheumatology, inflammatory bowel disease, multiple sclerosis, and diabetes). These specialty clinics are part of a large academic health campus, accept patients initially via referral for specialty care, then follow them for their chronic disease management. Inclusion criteria for the survey study were the following: patients who had (1) visited 1 of the 5 aforementioned clinics between January 2016 and July 2018, and (2) signed up for patient portal access. A total of 3421 patients met inclusion criteria for the survey during the study period.

A novel 30-question survey instrument (Multimedia Appendix 1) was developed by members of our research team using a modified Delphi [23] approach. Four overarching themes were identified: (1) satisfaction with the patient portal, (2) utility of the patient portal, (3) impact of the patient portal, and (4) demographic characteristics. Proposed questions were refined by the research team and then pretested with 6 members of our patient portal working group. After further refinement, which included changes and clarification of wording and question structure, the survey tool was piloted and sensibility-tested with a small group of patients, leading to a final survey with 5 (satisfaction, utility) to 12 (impact) questions per theme. The survey required about 10-15 minutes to complete.

During the sign-up process for the patient portal, patients consented to be contacted via their email address, which was routinely recorded as part of the sign-up process. Patient portal users were emailed a voluntary link to the survey which contained Likert scales, multiple-choice questions and free-text questions. The free-text answers were collated and thematically coded by two authors (TG, SA). Coding discrepancies were resolved by consensus. Surveys were administered at 2-week and 6-month time points relative to each participant’s first successful access to the patient portal, as this was felt to represent an adequate period of time for participants to access and become comfortable with using the patient portal. Respondents were made aware that their responses would not impact their clinical care, that no incentives were provided, and that their responses would only be studied in collated form. Respondents were permitted to skip any question they wished.

To determine the no-show rate, the numerator of scheduled appointments recorded as no-show was divided by the denominator of total scheduled appointments. No-show rates were calculated for users with an active patient portal account at the time of the visit and compared to rates for users attending the same clinics without a patient portal account at the time of the visit. Audit log analysis included attended and no-show visit rates, basic patient demographic data, and the patient’s contact email address.

Categorical data (eg, sex, age) were summarized with frequency distributions. Respondent demographic characteristics were compared to the wider user population with respect to sex and age. No-show rates were compared for patient visits where a patient portal account was active at the time and patient visits in the same clinic with no active patient portal account between November 2015 (just before the patient portal was introduced) and July 2018. Each respondent’s final answer across the 3 time points to questions of a more summative nature were included for statistical testing. Statistical differences were detected at a P value of less than .05. Chi-square testing for each of these comparisons was conducted using R statistical software (version 3.4.3, R Core Team, R Foundation for Statistical Computing). We included users who enrolled to use the system between May 2016 and May 2018. An additional 875 new users signed up for patient portal access between May and July of 2018. To avoid biasing the estimate of the proportions of users toward the early users, these users were included in the user population but not in survey analysis.

A total of 5629 electronic surveys were transmitted to 3421 patients between January 2016 and July 2018. The final administration of the survey was completed in May 2018 to allow users 2 months to respond to the survey. Overall, 957 patients responded 1916 times, and 28% (957/3421) of users receiving an invitation to participate replied to at least one survey. In addition to accessing their own health information, patient portal users reported using the patient portal for their children (36/957, 4%) or their spouse (131/957, 16%). Of 174,298 login sessions, 70% (122,648/174,298) were from a computer browser, 20% (34,205/174,298) were from an iPhone, and 10% (17,445) were from an Android device.

Table 1 describes user and respondent demographics and comfort with computer use; the majority of respondents (834/957, 87%) reported being either comfortable or completely comfortable using computers).

Patient portal users reported a high degree of usability and general satisfaction: 93% (891/957) of respondents felt the patient portal was easy to use, 83% (794/957) said it made communication more convenient, and 75% (716/957) indicated it saved time when scheduling an appointment.

Among survey respondents, 48% (460/957) stated the patient portal had helped them avoid a clinic visit. In February 2016, we added an additional question regarding emergency department or urgent care visit avoidance, and of users who answered this, 14% (26/188) answered that the patient portal allowed them to avoid such a visit.

There were 4 free-text questions available based on the themes of requested improvements, preferred features, how the patient portal helped avoid a health care visit (if applicable), and self-reported cost savings related to the patient portal (Table 2). Over two-thirds of respondents identified that access to laboratory and diagnostic imaging results was the most preferred feature of the patient portal. The most frequently reported concerns were related to a cumbersome sign-in process and a desire for more explanation of results. It should be noted that the patient portal was launched with an authentication method that was acknowledged to be overly complex at the time of implementation but was required for local technical and security reasons. Less common requests included a wider availability of data, including physician notes on the chart, an explanation of the meaning of the results, and the timeliness of the reports in the system.

We studied the proportion of patients who had a no-show to an appointment, as it related to patient portal user status. From January 2016 to July 2018, there were 19,968 no-shows across the 5 clinics that participated. Of these visits, 9021 did not have an active patient portal account at the time of the visit. The number of no-shows amongst these visits for patients without a patient portal was 858 (9.5%, range: 3.7%-16.6%). Of the remaining 10,947 visits for patients with an active patient portal account, 493 (4.5%) no-show visits were recorded (range: 3.1%-10.4%). No-show rates for patients with no patient portal were 11.9% (386/3257) for the specialty clinics and 8.2% (472/5764) for the family practice clinic, whereas the rates for patients with patient portal access were 4.9% (194/3943) and 4.3% (299/7004), respectively. This represented a 53% relative reduction in the no-show rate overall (P<.001).

In our study, patients were surveyed about their perceptions of an EMR-tethered patient portal. Patients had high general satisfaction, with over 90% reporting that it was easy to use, and almost half reporting that it saved them a medical visit. Respondents reported that access to results and advanced features, such as messaging and scheduling, improved communication with their care providers. Additionally, self-reported reductions in health system use implied improved utilization of scarce health system resources. Finally, objective data from clinic visits demonstrated an over 50% relative reduction in no-show rates.

We decided to focus on patient-centered indicators of patient portal utility rather than tracking health outcomes in our study. As with any technology, the benefits will accrue only if the technology itself is working and is being used as expected [24]. Although it is assumed that just having access to health information is of benefit, there is little evidence that access to lab tests will improve health outcomes. However, we believe that more advanced features, such as scheduling and secure messaging, in combination with access to test results and clinical notes are more likely to result in long-term health system benefits. In 2014, Goldzweig et al [17] performed a systematic review and found that there were a limited number of studies about patient portals, with heterogeneous designs making it difficult to draw strong conclusions about a new technology. Additionally, they found that improved outcomes (eg, for chronic disease such as diabetes, hypertension, and depression) tended to involve portal use in conjunction with case management [17]. They also found mixed data about the effect of portals on health care utilization and efficiency. Based on the success of the patient portal in our study, planning is underway to expand patient portal use across all AHS clinics. This expansion is targeted at the notion that patients having access to their own information in conjunction with the ability to interact with the health system digitally will contribute to increasing health system efficiency, including reductions in no-show rates and clinic and emergency department (ED) visits, increased patient satisfaction and empowerment and, in the longer term, improved health outcomes. The latter assumes that the patient portal is not only widely available, but actively being used as intended.

The 53% relative reduction in the no-show rate seen in patient portal users in the 5 pilot clinics may be widely relevant. Across the Edmonton zone of AHS, about 200,000 appointments are scheduled monthly. Of these, about 40,000 appointments are rescheduled, and 2000 appointments are considered no-shows. Patients who miss appointments often use emergency departments as sources of both primary and chronic care, driving up costs and straining hospital systems [23-25]. Missed appointments can compromise continuity and quality of care for both the patients who no-show and others who would have been scheduled in those appointment slots [25]. If the decreased no-show rates scale up to a wider set of users, patient portals may have significant positive impact on wait times and clinic efficiency. Access to scheduling functions in a patient portal may also be associated with decreased urgent care visits, although studies are not consistent [26,27]. In the current work, 2.7% of respondents reported being able to avoid one or more ED or urgent care visits, and 48% reported avoiding a clinic visit due to being able to communicate electronically. ED overcrowding is an ongoing and seemingly intractable problem; any strategy to safely decant these pressures in the Canadian context warrants further exploration.

In the near future, patients will expect web-based and mobile app–based access to their health data, and the ability to manage their health and interact with the health system digitally. In some jurisdictions, this access is not only related to tests and scheduling, but extends to transparent access into the complete medical record including progress notes [28]. Future studies will include interviews of care providers and patients to get a deeper understanding of the pros and cons of an EMR-tethered patient portal, its effect on clinic workflows, and the potential savings, whether that be in terms of time saved not waiting on hold during a phone call, or travel costs saved by replacing an inpatient visit with a secure message.

Our study evaluated a single patient portal in a large group of patients attending 1 of 5 clinics, in a single city, where the clinic leads were keen to implement it. It is not clear whether the results would be broadly generalizable. However, the study population did include both family medicine and specialty clinics, representing some diversity of needs and practice. Self-reported data is subject to recall bias, which we attempted to limit by sampling at different times after a user’s first successful patient portal access. The correlation between patient portal use and no-show rates may be partly the result of more diligent or engaged patients at the clinics participating in the study. Although we were able to compare survey respondent versus non-respondent demographic characteristics for all patients that had patient portal access, we did not have the ability to do the same for those with patient portal access versus those without. The analysis includes visits from non–patient portal users who eventually obtained patient portal access, which may underreport the effect size. The respondent population reported a high degree of comfort with computers, which may not hold true elsewhere, although it may reflect increasing competence with technology in general.

Early experience with an EMR-tethered patient portal showed decreased no-show rates for appointments, high patient satisfaction, and self-reported changes in health system use. Implications on the expanded use of patient portals requires further study in Canada.