This paper is in the following e-collection/theme issue:

Personal Health Records, Patient-Accessible Electronic Health Records, Patient Portals (565) Participatory Medicine & E-Patients (765) Voice Assistants (45) Influenza and Influenza-Like-Illness (ILI) Surveillance (214) Vaccination and Immunization in the Digital Age (363)

Published on in Vol 22, No 9 (2020): September

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/16373, first published .
Electronic Health Record Portal Messages and Interactive Voice Response Calls to Improve Rates of Early Season Influenza Vaccination: Randomized Controlled Trial

Electronic Health Record Portal Messages and Interactive Voice Response Calls to Improve Rates of Early Season Influenza Vaccination: Randomized Controlled Trial

Electronic Health Record Portal Messages and Interactive Voice Response Calls to Improve Rates of Early Season Influenza Vaccination: Randomized Controlled Trial

Authors of this article:

Jessica G Wijesundara1 Author Orcid Image ;   Mayuko Ito Fukunaga1, 2, 3 Author Orcid Image ;   Jessica Ogarek4 Author Orcid Image ;   Bruce Barton1 Author Orcid Image ;   Lloyd Fisher5, 6 Author Orcid Image ;   Peggy Preusse6 Author Orcid Image ;   Devi Sundaresan6 Author Orcid Image ;   Lawrence Garber2, 6 Author Orcid Image ;   Kathleen M Mazor2, 3 Author Orcid Image ;   Sarah L Cutrona1, 7 Author Orcid Image
Article Authors Cited by (15) Tweetations (3) Metrics

    Original Paper

    1Department of Population and Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA, United States

    2Meyers Primary Care Institute, Worcester, MA, United States

    3Department of Medicine, University of Massachusetts Medical School, Worcester, MA, United States

    4Center for Gerontology and Healthcare Research, Brown University, Providence, MA, United States

    5Department of Pediatrics, University of Massachusetts Medical School, Worcester, MA, United States

    6Reliant Medical Group, Worcester, MA, United States

    7Health Services Research & Development, Center of Innovation, Edith Nourse Rogers Memorial Hospital, Veterans Health Administration, Bedford, MA, United States

    Corresponding Author:

    Sarah L Cutrona, MD, MPH

    Health Services Research & Development, Center of Innovation

    Edith Nourse Rogers Memorial Hospital

    Veterans Health Administration

    200 Springs St

    Building 70

    Bedford, MA, 01730

    United States

    Phone: 1 508 856 4046

    Email: Sarah.Cutrona@umassmed.edu


    Background: Patient reminders for influenza vaccination, delivered via an electronic health record patient portal and interactive voice response calls, offer an innovative approach to engaging patients and improving patient care.

    Objective: The goal of this study was to test the effectiveness of portal and interactive voice response outreach in improving rates of influenza vaccination by targeting patients in early September, shortly after vaccinations became available.

    Methods: Using electronic health record portal messages and interactive voice response calls promoting influenza vaccination, outreach was conducted in September 2015. Participants included adult patients within a large multispecialty group practice in central Massachusetts. Our main outcome was electronic health record–documented early influenza vaccination during the 2015-2016 influenza season, measured in November 2015. We randomly assigned all active portal users to 1 of 2 groups: (1) receiving a portal message promoting influenza vaccinations, listing upcoming clinics, and offering online scheduling of vaccination appointments (n=19,506) or (2) receiving usual care (n=19,505). We randomly assigned all portal nonusers to 1 of 2 groups: (1) receiving interactive voice response call (n=15,000) or (2) receiving usual care (n=43,596). The intervention also solicited patient self-reports on influenza vaccinations completed outside the clinic. Self-reported influenza vaccination data were uploaded into the electronic health records to increase the accuracy of existing provider-directed electronic health record clinical decision support (vaccination alerts) but were excluded from main analyses.

    Results: Among portal users, 28.4% (5549/19,506) of those randomized to receive messages and 27.1% (5294/19,505) of the usual care group had influenza vaccinations documented by November 2015 (P=.004). In multivariate analysis of portal users, message recipients were slightly more likely to have documented vaccinations when compared to the usual care group (OR 1.07, 95% CI 1.02-1.12). Among portal nonusers, 8.4% (1262/15,000) of those randomized to receive calls and 8.2% (3586/43,596) of usual care had documented vaccinations (P=.47), and multivariate analysis showed nonsignificant differences. Over half of portal messages sent were opened (10,112/19,479; 51.9%), and over half of interactive voice response calls placed (7599/14,984; 50.7%) reached their intended target, thus we attained similar levels of exposure to the messaging for both interventions. Among portal message recipients, 25.4% of message openers (2570/10,112) responded to a subsequent question on receipt of influenza vaccination; among interactive voice response recipients, 72.5% of those reached (5513/7599) responded to a similar question.

    Conclusions: Portal message outreach to a general primary care population achieved a small but statistically significant improvement in rates of influenza vaccination (OR 1.07, 95% CI 1.02-1.12). Interactive voice response calls did not significantly improve vaccination rates among portal nonusers (OR 1.03, 95% CI 0.96-1.10). Rates of patient engagement with both modalities were favorable.

    Trial Registration: ClinicalTrials.gov NCT02266277; https://clinicaltrials.gov/ct2/show/NCT02266277

    J Med Internet Res 2020;22(9):e16373

    doi:10.2196/16373

    Keywords

    electronic health records; influenza vaccination; patient care; patient engagement 


    Influenza infections contribute to increased health care costs and loss of productivity and can lead to serious complications and even death [1]. Effective strategies for the prevention of influenza are of critical importance as we enter the 2020-2021 influenza season. The confluence of the upcoming influenza season and the ongoing coronavirus disease 2019 (COVID-19) pandemic is expected to place additional stress on our health care system, fueled in part by similarities in presenting complaints for these two illnesses, as well as by the potential for increased risk of poor outcomes in patients co-infected with COVID-19 and influenza [2-4].

    An estimated 5%-20% of the US population contracts influenza every year, with several hundred thousand people hospitalized annually due to influenza-related complications [5]. Estimates of annual influenza and pneumonia-associated deaths over the past decade reached as high as 61,000 in the 2017-2018 season [6-12]. According to Centers for Disease Control and Prevention (CDC) estimates, during the 2018-2019 influenza season, vaccinations prevented approximately 4.4 million flu illnesses, 58,000 hospitalizations, and 3500 deaths [13,14]. Since it takes approximately 2 weeks for antibodies to develop in response to influenza vaccination, the CDC recommends getting vaccinated before flu begins to spread within a community, by the end of October [15].

    Despite widespread publicity promoting influenza vaccination, vaccines are underutilized [16-19]. In 2017, national vaccination coverage among adults for influenza was 37.1% [20], while the Healthy People 2020 target was 70% [21]. The CDC estimated flu vaccination coverage among adults aged ≥18 years as of mid-November 2018 was 44.9% [22]; at the end of the 2018-2019 influenza season, national influenza vaccination coverage was 45.3%[23].

    Clinical decision support has been shown to improve health outcomes by supporting the delivery of timely, evidence-based and guideline-concordant medical care, including annual vaccinations [24-26]. Clinical decision support includes computerized reminders both to providers and patients. Many health systems effectively use provider-directed clinical decision support, including noninterruptive or interruptive (pop-up) alerts, reminding providers of recommended prevention or screening measures [27,28]. While frequently effective, provider-directed clinical decision support is subject to important limitations. If alerts are triggered by erroneous or incomplete electronic health record data, or if providers experience alert fatigue from an overwhelming number of notifications, alerts may be ignored or overridden [29-32]. Considering these challenges, and in the setting of nationwide adoption of electronic patient portals, patient-directed clinical decision support delivered via a patient portal offers an innovative approach to the promotion of timely influenza vaccination.

    Electronic patient portals are secure websites that provide patients with 24-hour online access to limited electronic health record information. A portal provides patients with a personal health record that is tethered to their electronic health record. Accessible information within a tethered portal varies by health system but may include vaccinations, laboratory results, problem lists, allergies, and information from recent doctor visits or hospitalizations [33,34]. A core function of portals is secure messaging—electronic communication with the physician or health care team [35]. Patient portals have the potential to improve patient-provider communication, improve medication adherence, decrease office visits, increase self-management of disease and disease awareness, increase use of preventative medicine, and increase inclusion of patients in medical decision making [36-38]. Previous patient outreach interventions have been shown to improve rates of vaccination completion and have been tested using multiple options including mailed letters, postcards, live phone calls, automated phone messages, and combination postcard/phone-based [39,40]. Few studies have tested the use of patient-directed vaccination reminders sent via patient portals [25,26,41].

    We conducted a randomized controlled trial aimed at improving rates of influenza vaccination among eligible adult patients in a large multispecialty group practice in central Massachusetts. We used electronic health record patient portal messages as well as interactive voice recognition calls to (1) promote early season influenza vaccination completion and (2) solicit patient self-report on vaccinations completed outside the clinic.


    Study Objectives

    The overarching goal of this study was to improve rates of early influenza vaccination (by the end of October) among eligible adults in an outpatient population. Our primary objective was to determine whether our outreach increased completion of influenza vaccinations and, if so, whether one mode of outreach was most effective. Additional objectives were to improve documentation of influenza vaccinations administered outside the practice by inviting patient self-report (improving the accuracy of existing decision support tools) and to track process measures (eg, rates of portal message opening and interactive voice response call answering).

    Study Design

    We conducted a nonblinded randomized controlled intervention (NCT02266277) at a large multispecialty medical group in central Massachusetts. Previously, our team developed and tested interactive voice response and portal outreach which we targeted to patients who had no documented vaccination 2 months after the start of the season [25,26]; this study adapted our previous approach, targeting a broader population in early September in order to promote early vaccination and provide information on September and October flu clinic dates.

    Using a computer-generated randomization table, we assigned all active portal users to receive either a portal message promoting influenza vaccination, listing upcoming clinics, and offering online scheduling of vaccination appointments (n=19,506) or usual care with no portal message (n=19,505). Separately, we randomized all portal nonusers to receive either an interactive voice response call (n=15,000) or usual care with no interactive voice response call (n=43,596) (Figure 1). For portal users only, after the conclusion of the study and assessment of outcomes on November 3, messages were sent to the usual care group if they still did not have an electronic health record–documented influenza vaccination for the 2015-2016 season (n=14,118). The cost of calls prevented us from being able to send interactive voice response messages to the usual care group in the portal nonusers.

    The study was reviewed and approved in 2014 by the Reliant Medical Group institutional review board; due to administrative changes, oversight was transferred to the University of Massachusetts institutional review board in 2015. A waiver for informed consent for patient outreach was approved by these institutional review boards. Patients were not compensated for their participation. The principal investigator (SC) oversaw the trial and data analysis.

    ‎
    Figure 1. CONSORT (Consolidated Standards of Reporting Trials) Diagram describing randomized controlled trial to improve rates of early season influenza vaccinations using EHR portal messages and Interactive Voice Response calls.
    View this figure

    Study Population

    Eligibility Criteria

    Patients were eligible for the study if they (1) had a primary care provider at the medical group during the 12 months prior to randomization; (2) were aged ≥18 on the date of randomization; (3) had a recent office visit or telephone encounter with an internal medicine practitioner or family practitioner (defined as having had an office visit, phone encounter, consult, or complete physical exam within the 12 months prior to randomization). This requirement was intended to minimize inclusion of patients who had moved to another practice but whose names were retained in the medical group records. To ensure capture of patients transitioning from pediatric to adult care, the recent office visit could also be with a pediatrician.

    A patient was eligible for inclusion in the electronic patient portal portion of the trial if the patient was an active user, which was defined as having an activated portal with a log-in at least once in the year preceding randomization.

    Exclusion Criteria

    For both interactive voice response and portal outreach, patients were excluded if there was electronic health record documentation of an allergy to influenza vaccines. For interactive voice response calls only, exclusion criteria also included the presence of any of the following on the date of randomization: (1) electronic health record documentation of influenza vaccination completion in the 2014-2015 influenza season (or documented influenza vaccination after the end of the 2014-2015 influenza season but before the start of the 2015-2016 season); (2) no listed phone number.

    Study Procedures

    This study consisted of interactive voice response or portal-based outreach. Qualitative interviews conducted with patients, physicians, nurses, and staff informed the development of our outreach material [25,26].

    Electronic Patient Portal Intervention

    We designed an outgoing secure portal message to be sent via patient portal to patients randomized to the portal message arm (Multimedia Appendix 1). Portal message content appeared in letter format with the signature line reflecting the name of the patient’s primary care provider. Portal messages were delivered through standard channels used for portal-based correspondence between the medical group’s health care providers and patients (ie, a generic message that contained neither personal health information nor any reference to vaccinations was delivered to patient’s email account; the message prompted patients to log in to secure portal account via hyperlink). Once logged in to the portal accounts, patients clicked on a message labeled “A Message from Your Primary Care Provider” to view the outreach message. The outreach message included information about upcoming flu clinics for September and October 2015.

    Unique to the portal message (compared to interactive voice response phone messages) was the option of direct online scheduling of appointments for influenza vaccination. Information about accessing the CDC vaccination website appeared within the body of the portal message as a hyperlink (and was conveyed verbally in the interactive voice response script). Opportunities to report external influenza vaccinations and to report intent to get vaccinated matched the interactive voice response call content.

    Portal Message Delivery

    Messages were sent out to 500 to 1500 patients daily over a period of 15 days in September 2015 in order to reduce the risk of being blocked by network bulk-spam filters.

    Interactive Voice Response Call Intervention

    Interactive voice response calls appeared on caller ID as originating from the medical group. This is consistent with identification of interactive voice response calls used for appointment reminders at the time of the study. Combining voice response with branching logic, calls elicited patient self-reports of influenza vaccinations completed outside the medical group (Multimedia Appendix 2). For patients reporting no influenza vaccination completed, calls included information about upcoming flu clinics for September and October 2015. Patients reporting no influenza vaccination completed were also asked whether they intended to get vaccinated. Patients reporting that they were unsure or did not intend to get vaccinated were asked further questions on specific reasons why they did not plan to get a flu vaccination.

    Interactive Voice Response Call Delivery

    Intervention interactive voice response calls, initiated on September 11, 2015, began by confirming that the person answering the phone was the intended patient recipient of the call. If voicemail was encountered or if the person reached was someone other than the patient, the interactive voice response system left a message asking patients to call back and provided an inbound call line number. The last interactive voice response outbound calls were placed on September 25, 2015. The inbound call line was maintained throughout the duration of outgoing calls and for 3 weeks after the final outgoing call was placed; patients who called this number from the phone number of record heard the interactive voice response call script in its entirety, beginning with questions confirming the identity of the caller.

    Study Outcomes

    Primary Outcome

    Our primary outcome was percentage of eligible patients with influenza vaccinations documented in the electronic health record as of November 3, 2015. We pulled data on vaccination rates as of this date, chosen in order to assess the impact of early outreach on completion of early immunization. Immunizations captured solely through the patient portal questionnaire or through the interactive voice response were excluded from the primary analysis in order to enable comparison with the control groups. The origins of all entered influenza vaccinations were tracked to allow our team to distinguish between sources of information on completed vaccinations.

    Process Measures and Additional Outcomes of Interest

    For portal messages, we calculated (1) percentage of recipients who logged in to the patient portal during early flu season (through November 3, 2015), (2) percentage of recipients who opened messages during early flu season, and (3) percentage of recipients who completed questionnaires. We tracked self-reports (via the portal) of influenza vaccinations completed. We also tracked patient-reported intent to get an influenza vaccination during the 2015-2016 flu season.

    For interactive voice response calls, we calculated (1) percentage of recipients reached and (2) percentage of recipients who completed the calls by responding to questions. We tracked self-report (via interactive voice response) of influenza vaccinations completed. We also tracked patient-reported intent to get an influenza vaccination during the 2015-2016 flu season.

    Sample Size

    With our proposed sample size, power calculations based on estimates of baseline vaccination rates indicated that 4286 participants per arm would give 80% power to detect a 3% improvement in influenza vaccination rates between groups (α=.05; 2-sided).

    Statistical Methods

    Primary Outcome

    To determine the impact of our interventions on early vaccination rates for the 2015-2016 influenza season, we calculated frequencies and performed intention-to-treat bivariate analyses of randomized patients, assessing whether vaccination completion was associated with group assignment. Due to different rates of electronic health record–recorded vaccination measured at baseline (in 2014) between portal users (35.9%) and nonusers (25%), and due to the differences in intervention (portal versus interactive voice response call), analyses for these groups were conducted separately.

    We then performed multivariate logistic regression analyses, adjusting for demographic and practice-level covariates, we modeled the odds of receiving an influenza vaccination in the 2015-2016 influenza season.


    Baseline Characteristics

    Baseline characteristics of both patient portal users and portal nonusers are reported in Table 1. Baseline characteristics were similar among portal users and portal nonusers. However, compared to the portal nonusers, portal users were more likely to be women, older, and have a higher level of health care utilization.

    Table 1. Baseline characteristics of participants.
    CharacteristicsAll (N=97,607), n (%)Portal usersPortal nonusers



    		Portal message (n=19,506), n (%)Usual care (n=19,505), n (%)Call (n=15,000), n (%)Usual care (n=43,596), n (%)
    Sex





    		Female53,250 (54.5)12,230 (62.7)12,249 (62.8)7325 (48.8)21,446 (49.2)

    		Male44,349 (45.4)7275 (37.3)7256 (37.2)7672 (51.1)22,146 (50.8)

    		Missing8 (0.01)1 (0.0)0 (0.0)3 (0.0)4 (0.0)
    Age





    		18-3430,556 (31.3)4508 (23.1)4597 (23.6)5615 (37.4)15,836 (36.3)

    		35 - 4925,579 (26.2)5096 (26.1)5062 (26.0)3896 (26.0)11,525 (26.4)

    		50 - 6426,495 (27.1)6275 (32.2)6224 (31.9)3547 (23.6)10,449 (24.0)

    		65-748593 (8.8)2419 (12.4)2388 (12.2)965 (6.4)2821 (6.5)

    		75+6384 (6.5)1208 (6.2)1234 (6.3)977 (6.5)2965 (6.8)
    Race





    		White66,020 (67.6)14,180 (72.7)14,223 (72.9)9608 (64.1)28,009 (64.2)

    		Black3417 (3.5)456 (2.3)440 (2.3)630 (4.2)1891 (4.3)

    		Asian3502 (3.6)737 (3.8)715 (3.7)528 (3.5)1522 (3.5)

    		American Indian or Alaska Native1021 (1.1)140 (0.7)144 (0.7)194 (1.3)543 (1.2)

    		Other10,366 (10.6)1672 (8.6)1745 (8.9)1769 (11.8)5180 (11.9)

    		Missing13,281 (13.6)2321 (11.9)2238 (11.5)2271 (15.1)6451 (14.8)
    Health care utilization level





    		Had office visita80,748 (82.7)17,896 (91.7)17,908 (91.8)11,609 (77.4)33,335 (76.5)

    		Did not have office visit16,859 (17.3)1610 (8.3)1597 (8.2)3391 (22.6)10,261 (23.5)

    a12 months prior to randomization.

    Portal Users

    Among portal users, 28.4% (5549/19,506) of message recipients and 27.1% (5294/19,505) of the usual care group had documentation in their electronic health records that they received that influenza vaccinations on or before November 3, 2015 (P=.004). Portal users who received the messages were significantly more likely to have received the influenza vaccination compared to the usual care group (odds ratio [OR] 1.07, 95% CI 1.02-1.12). This finding was consistent even after adjusting for age, race, sex, and health care utilization (OR 1.07, 95% CI 1.02-1.12) (Table 2).

    Table 2. Likelihood of receiving an early-season influenza vaccination.
    RecipientsnUnadjustedAdjusteda


    		ORb95% CIOR95% CI
    Portal messagec39,0111.07(1.02, 1.12)1.07(1.02, 1.12)
    Interactive voice response callc58,5961.03(0.96, 1.10)

    		1.03(0.96, 1.11)

    aAdjusted for age, sex, race, and health care utilization level (where utilization was defined as office visit, phone encounter, consult, or complete physical exam within the 12 months prior to randomization).

    bOR: odds ratio.

    cThe reference is usual care.

    Portal Nonusers

    Among portal nonusers, 8.4% (1262/15,000) of call recipients and 8.2% (3586/43,596) of usual care recipients received vaccinations (P=.47). Bivariate and multivariate analysis showed nonsignificant differences in influenza vaccination rates between intervention and usual care groups (Table 2).

    Process Measures

    Portal Message

    Among patient portal message recipients, 71.2% (13,862 recipients out of 19,479 to whom the message was sent) logged in to the patient portal on or after the date of message delivery through the end of early flu season (September 9, 2015 to November 3, 2015). Messages were opened by 51.9% (10,112/19,479) recipients; 13.2% (2570/19,479) responded to the first question asking if they received a flu vaccination on or after August 1, 2015, 2.0% (386/19,479) reported already receiving a vaccination and 11.2% (2176/19,479) responded to a second question assessing their intention to receive a flu vaccination during the flu season (asked only of those who were not already vaccinated) (Table 3).

    Of those opening messages, 25.4% (2570/10,112) responded to our question on receipt of influenza vaccination, 3.8% (386/10,112) reported already receiving vaccinations and 21.5% (2176/10,112) responded to our question on whether they planned to get vaccinated.

    Table 3. Process measures and self-reported influenza vaccinations for portal message recipients.
    ActionPortal users, n (%)
    Randomized19,506
    Message senta19,479 (100)
    Logged in to patient portal13,862 (71.2)
    Opened message10,112 (51.9)
    Responded to “Have you received a flu vaccination on or after August 1, 2015?”2570 (13.2)
    Reported receiving a flu vaccination386 (2.0)
    Responded to “Do you plan to get a flu vaccination this flu season?”2176 (11.2)
    Reported that they planned to get a flu vaccination1814 (9.3)

    a27 patients were no longer portal users by the time we sent the messages (due to change in medical record numbers, invalid patient portal IDs, etc).

    Interactive Voice Response Call

    Among interactive voice response call recipients, 50.7% of patients were reached (7599/14,984); 36.8% (5513/14,984) responded to the first question asking if they received a flu vaccination on or after August 1, 2015, 3.8% (575/14,984) reported receiving their vaccination and 24.1% (3613/14,984) responded to a second question assessing their intention to receive a flu vaccination during the flu season (asked only of those who were not already vaccinated) (Table 4).

    Table 4. Process measures and self-reported influenza vaccinations for interactive voice response call recipients.
    Action Call recipients, n (%)
    Randomized14,999
    Call attempteda14,984 (100)
    Target reached (inbound and outbound calls)7599 (50.7)
    Responded to “Have you received a flu vaccination on or after August 1, 2015?”5513 (36.8)
    Reported receiving a flu vaccination575 (3.8)
    Responded to “Do you plan to get a flu vaccination this flu season?”3613 (24.1)
    Reported that they planned to get a flu vaccination1415 (9.4)

    a15 patients had invalid/blank phone numbers, or the patient was on a “Do Not Call” list, by the time we placed the calls.

    Of those reached, 72.5% (5513/7599) responded to our question on receipt of influenza vaccination, 7.6% (575/7599) reported already receiving a vaccination, and 47.5% (3613/7599) responded to our question on whether they planned to get vaccinated.


    Principal Results

    Our study showed a clinically small but statistically significant improvement in completion of early season influenza vaccinations among those randomized to receive outreach via patient portal, compared to a usual care control group (OR 1.07, 95% CI 1.02-1.12). There was no significant increase of early season influenza vaccinations among those randomized to receive an interactive voice response call (OR 1.03, 95% CI 0.96-1.10).

    This outreach was designed to deliver a relatively time-sensitive message (eg, reminding patients of the importance of influenza vaccination and alerting them of upcoming vaccination clinics in September and October), and as such, represents a successful, brief patient engagement effort. We attained greater than 50% opening rates for portal messages and reached the targeted patient on over 50% of interactive voice response calls, achieving similar levels of exposure to the messaging for the two interventions. Among portal message recipients, more than one quarter of those who opened the message responded to our subsequent question on whether they had received or intended to receive their influenza vaccination. Among interactive voice response call recipients, close to three-quarters of those reached responded to the question.

    A small number of patients reported influenza vaccinations that had been completed in the community; data on patient-reported vaccinations performed in the community were then uploaded into the electronic health record in order to improve the accuracy of existing influenza vaccination alerts directed at primary care providers. The relatively small number of self-reported flu vaccinations (961/20,978 participants across the portal message intervention and the interactive voice response intervention arms combined) has several possible explanations. It is possible that participants who had already been vaccinated found the outreach less relevant; and therefore, chose not to engage (eg, chose not to open the message or chose to hang up the phone). Relatively low rates of self-reported vaccination completion may also be attributable to the timing of our outreach—intentionally positioned at the start of the flu season. In contrast, our past work [26] captured much higher proportions of self-reported vaccinations (2591/20,000 intervention patients) when participants were approached several months past the start of flu season, which highlights the importance of asking patients to self-report vaccinations administered outside the clinic.

    Although these “discovered” immunizations were incorporated into the electronic health record, they were not counted in the primary analysis; this allowed comparison with the usual care groups who were not queried for outside immunizations. While recognizing that the actual immunization rate was higher than the documented rate, any differences in the documented rate between intervention and control groups can reasonably be attributed to increased vaccination rates. Data on intention to be vaccinated were also elicited. Studies have shown that stating intent to complete a behavior can enhance likelihood of follow-through [42,43], thus it is possible that engaging patients in this manner may have contributed to the modest success of our intervention.

    Once designed, portal messages (sent out in several batches) required minimal staff time to deliver. Our messages were designed to be easily adapted for use in future years; our messages have already been adapted and implemented by our medical center to target high-risk populations (eg, children with asthma).

    Comparison With Prior Work

    There have been multiple studies [44-47] documenting success in use of patient-directed reminders for influenza vaccination. Effective reminders described include letters, telephone invitations, phone calls from a peer, tailored communications, customized letters/phone calls, and client-based appraisals [44]. Text message reminders have been shown to be successful for influenza vaccination reminders in pediatric and adolescent populations [45], in patients with rare diseases [46], and among high-risk patients [47].

    While portal-based outreach directed at patients has been described for a variety of preventive measures [48], few studies have tested use of patient-directed influenza vaccination reminders sent via patient portals; one study [49] that has done so focused exclusively on untethered (ie, personally controlled) patient health records among a university student and staff population in Australia, yielding a 6.7% greater likelihood of influenza vaccination among users of the personal health records compared to those randomized to a 6-month waitlist for portal activation [49]. In our past work [25,26], we tested similar portal and interactive voice response outreach messages using a factorial design and targeting a population more likely to be nonadherent to vaccination guidelines (patients who had no documented vaccination 2 months after the start of the flu season). In that study [25,26], we found small but statistically significant improvements in influenza vaccination rates among recipients of either outreach method compared to usual care (portal message alone (OR 1.20, 95% CI 1.06-1.35); interactive voice response call alone (OR 1.15, 95 % CI 1.02-1.30); both messages (OR 1.29, 97.5% CI 1.13-1.48). In their recent randomized trial, Szilagyi et al [41] studied the effect of 1, 2, or 3 patient portal reminders on influenza vaccination rates among 164,205 patients in 52 primary care practices. They noted small but statistically significant improvements in vaccination rates with an attenuating effect of repeated messaging (37.5% for those receiving no reminders; 38.0% for 1 reminder; 38.2% for 2 reminders, and 38.2% for 3 reminders). Higher overall rates of vaccination were documented in their trial in comparison to ours; this is likely due in part to the study’s inclusion of children (who have a higher rate of influenza vaccination than that of the overall adult population) as well as their measurement of vaccination rates at the end of the influenza season (March 31).

    The rates of patient engagement (measured by message opening and subsequent action) demonstrated by our patient portal recipients were comparable to those from previously published patient portal outreach studies. In our previous influenza outreach intervention [25,26] (in a population unvaccinated 2 months into the flu season), messages were opened by slightly more than half of message recipients and interactive voice response targets were reached in just over 60%. In that prior study, among those who opened portal messages, 28.6% responded to subsequent questions. Among those reached via interactive voice response call, 78.3% responded to questions. In their patient portal intervention aimed at improving influenza vaccination rates, Szilagyi and colleagues [41] found opening rates of 52.9% among patients receiving a single portal message (increasing to 55.9% in a 2-reminder group and 58.8% in a 3-reminder group).

    Fischer and colleagues [50] found that among patients randomized to active reminders about multiple health maintenance services (eg, hemoglobin A1c testing, lipids, etc), nearly 65% of patients logged in to the portal after receiving the first of several messages. In a study [51] of colorectal cancer screening reminders delivered via patient portal, among the 552 patients randomized to receive messages, 54% viewed the message and 9% performed a suggested web-based risk assessment tool. In a study describing the reach and feasibility of an interactive lung cancer screening decision aid delivered by patient portal, Dharod and colleagues [52] found that 86% of lung cancer screening eligible patients identified by an electronic health record algorithm to receive a patient portal message read the message, 40% then visited a web-based decision aid for lung cancer screening, and 35% completed questionnaires to determine their eligibility for lung cancer screening.

    Limitations

    Our study had several limitations. Despite responding to inquiries on flu vaccinations receipt at higher rates, the interactive voice response group did not show a significant impact on vaccination rates (while the portal message group showed a small statistically significant improvement in vaccination rates). This discrepancy could be due in part to differences in the baseline characteristics of portal nonusers compared to portal users. For several reasons, interactive voice response call recipients may have been more resistant to completion of influenza vaccination than portal message recipients. Portal nonusers were less likely to have had an office visit in the previous 12 months and might have been less actively engaged in their health care overall. In addition, eligibility for the interactive voice response study was intentionally more stringent, with patients included only if they lacked documentation of an influenza vaccination in the previous flu season. This study design choice was made with guidance from the medical center where our study was implemented and was intended to make available the limited resource of interactive voice response calls to the broadest possible number of nonadherent patients (our study covered the cost for only 15,000 calls but we had no limit on the number of portal messages that we could send). It is possible that the interactive voice response calls might have yielded a greater impact if we had not opted to employ this more stringent eligibility criterion.

    Conclusions

    There are compelling reasons to use existing functionality within electronic health record–tethered portals to promote influenza vaccination. For vaccination outreach, data recorded in the electronic health record through routine care delivery can inform real-time identification of unvaccinated populations. Portal-based outreach can be more cost-effective than phone calls or mailings and easier to implement than a new app; simple messages can be sent out by office staff without additional informatics expertise. Studies [37,38,53,54] show that patient portals can enhance patient empowerment and sense of autonomy, enhance patient engagement, improve medication adherence, decrease office visits, increase self-management of disease and disease awareness, increase use of preventive medicine, and increase inclusion of patients in medical decision making.

    Our parallel interventions to patient portal users and portal nonusers allowed us to assess the impact of outreach supporting influenza vaccination in both groups. Our study demonstrated a small improvement in influenza vaccination rates among portal message recipients, and successful patient engagement in both portal message recipients and interactive voice response call recipients. This method is readily applicable to current practice.

    As we simultaneously face the upcoming influenza season and the ongoing COVID-19 pandemic, expanding influenza vaccination coverage in ambulatory populations can decrease the strain on our overtaxed health system and may help avert poor outcomes in patients at risk for co-infection with influenza and COVID-19.

    Our findings of a small but significant improvement in influenza vaccination rates resulting from portal-based outreach represent an important contribution to the national conversation on caring for and protecting our patients in the upcoming months and for years to come.

    Acknowledgments

    This study was funded by Pfizer Independent Grants for Learning and Change. The authors designed and implemented the study with full oversight and responsibility for data collection, analysis, and manuscript preparation. Our study team acknowledges the contributions made by Eliza, an HMS Company, toward the design and implementation of the interactive voice response calls. A portion of MIF’s work was supported by the National Heart, Lung, and Blood Institute through Grant 1K12HK138049-01. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Example of portal outreach message.

    PDF File (Adobe PDF File), 237 KB
    ‎
    Multimedia Appendix 2

    Example of interactive voice response call script.

    PDF File (Adobe PDF File), 159 KB
    1. Molinari NM, Ortega-Sanchez IR, Messonnier ML, Thompson WW, Wortley PM, Weintraub E, et al. The annual impact of seasonal influenza in the US: measuring disease burden and costs. Vaccine 2007 Jun 28;25(27):5086-5096. [CrossRef] [Medline]
    2. Gostin LO, Salmon DA. The Dual Epidemics of COVID-19 and Influenza: Vaccine Acceptance, Coverage, and Mandates. JAMA 2020 Jul 28;324(4):335-336. [CrossRef] [Medline]
    3. Antony SJ, Almaghlouth NK, Heydemann EL. Are coinfections with COVID-19 and influenza low or underreported? An observational study examining current published literature including three new unpublished cases. J Med Virol 2020 Jun 12:26167. [CrossRef] [Medline]
    4. Yue H, Zhang M, Xing L, Wang K, Rao X, Liu H, et al. The epidemiology and clinical characteristics of co-infection of SARS-CoV-2 and influenza viruses in patients during COVID-19 outbreak. J Med Virol 2020 Jun 12:26163 [FREE Full text] [CrossRef] [Medline]
    5. NIH fact sheets - influenza. NIH. 2011.   URL: https:/​/archives.​nih.gov/​asites/​report/​09-09-2019/​report.nih.gov/​nihfactsheets/​ViewFactSheet44bd.​html?csid=133&key=I#I [accessed 2019-09-27]
    6. National CFHS. Health, United States. Health, United States 2015;With special feature on racial and ethnic health disparities. Hyattsville (MD):104-112. [CrossRef]
    7. Kochanek KD, Murphy SL, Xu J. Deaths: Final Data for 2011. Natl Vital Stat Rep 2015 Jul 27;63(3):1-120 [FREE Full text] [Medline]
    8. Kochanek KD, Xu J, Murphy SL, Miniño AM, Kung H. Deaths: final data for 2009. Natl Vital Stat Rep 2011 Dec 29;60(3):1-116 [FREE Full text] [Medline]
    9. Murphy SL, Kochanek KD, Xu J, Heron M. Deaths: Final Data for 2012. Natl Vital Stat Rep 2015 Aug 31;63(9):1-117 [FREE Full text] [Medline]
    10. Murphy SL, Xu J, Kochanek KD. Deaths: final data for 2010. Natl Vital Stat Rep 2013 May 8;61(4):1-117 [FREE Full text] [Medline]
    11. Xu J, Murphy SL, Kochanek KD, Bastian BA. Deaths: Final Data for 2013. Natl Vital Stat Rep 2016 Feb 16;64(2):1-119 [FREE Full text] [Medline]
    12. Prevention CDC. Estimated Influenza Illnesses, Medical visits, Hospitalizations, and Deaths in the United States - 2017-2018 influenza season. 2019.   URL: https://www.cdc.gov/flu/about/burden/2017-2018.htm [accessed 2019-09-27]
    13. Rolfes MA, Flannery B, Chung J, O'Halloran A, Garg S, Belongia EA, U.S. Flu VE Network‚ the Influenza Hospitalization Surveillance Network (FluSurv-NET)‚the Assessment Branch‚ Immunization Services Division‚ Centers for Disease ControlPrevention. Effects of Influenza Vaccination in the United States during the 2017-2018 Influenza Season. Clin Infect Dis 2019 Feb 02:1845-1853. [CrossRef] [Medline]
    14. Prevention CDC. Estimated Influenza Illnesses, Medical visits, Hospitalizations, and Deaths in the United States - 2018-2019 influenza season. 2020.   URL: https://www.cdc.gov/flu/about/burden/2018-2019.html [accessed 2019-09-27]
    15. Frequently asked flu questions 2018-2019 influenza season. Centers for Disease Control and Prevention. 2019.   URL: https://www.cdc.gov/flu/about/season/flu-season-2018-2019.htm
    16. National Vaccine Advisory Committee. A pathway to leadership for adult immunization: recommendations of the National Vaccine Advisory Committee: approved by the National Vaccine Advisory Committee on June 14, 2011. Public Health Rep 2012;127 Suppl 1:1-42 [FREE Full text] [Medline]
    17. Appel A. Improving adult immunization rates: overcoming barriers. Am Fam Physician 2011 Nov 01;84(9):977-978 [FREE Full text] [Medline]
    18. Bean-Mayberry B, Yano EM, Mor MK, Bayliss NK, Xu X, Fine MJ. Does sex influence immunization status for influenza and pneumonia in older veterans? J Am Geriatr Soc 2009 Aug;57(8):1427-1432 [FREE Full text] [CrossRef] [Medline]
    19. Vaughn JA, Miller RA. Update on immunizations in adults. Am Fam Physician 2011 Nov 01;84(9):1015-1020 [FREE Full text] [Medline]
    20. Estimates of influenza vaccination coverage among adults - United States, 2017-18 flu season. Centers for Disease Control and Prevention. 2018.   URL: https://www.cdc.gov/flu/fluvaxview/coverage-1718estimates.htm [accessed 2019-06-14]
    21. HealthyPeople.gov. Immunization and infectious diseases data details.   URL: https://www.healthypeople.gov/node/3527/data-details [accessed 2019-10-17]
    22. Early-season flu vaccination coverage - United States, November 2018. Centers for Disease Control and Prevention. 2018.   URL: https://www.cdc.gov/flu/fluvaxview/nifs-estimates-nov2018.htm [accessed 2019-06-14]
    23. Prevention CDC. Flu Vaccination Coverage, United States, 2018-19 Influenza Season. 2019.   URL: https://www.cdc.gov/flu/fluvaxview/coverage-1819estimates.htm [accessed 2019-09-27]
    24. Clinical decision support: more than just 'alerts' tipsheet. Clinical decision support: more than just 'alerts' tipsheet. US Centers for Medicare & Medicaid Service.   URL: https:/​/www.​cms.gov/​regulations-and-guidance/​legislation/​EHRincentiveprograms/​downloads/​clinicaldecisionsupport_tipsheet-.​pdf
    25. Cutrona SL, Sreedhara M, Goff SL, Fisher LD, Preusse P, Jackson M, et al. Improving Rates of Influenza Vaccination Through Electronic Health Record Portal Messages, Interactive Voice Recognition Calls and Patient-Enabled Electronic Health Record Updates: Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2016 May 06;5(2):e56 [FREE Full text] [CrossRef] [Medline]
    26. Cutrona SL, Golden JG, Goff SL, Ogarek J, Barton B, Fisher L, et al. Improving Rates of Outpatient Influenza Vaccination Through EHR Portal Messages and Interactive Automated Calls: A Randomized Controlled Trial. J Gen Intern Med 2018 May;33(5):659-667 [FREE Full text] [CrossRef] [Medline]
    27. Cutrona SL, Fouayzi H, Burns L, Sadasivam RS, Mazor KM, Gurwitz JH, et al. Primary Care Providers' Opening of Time-Sensitive Alerts Sent to Commercial Electronic Health Record InBaskets. J Gen Intern Med 2017 Nov;32(11):1210-1219. [CrossRef] [Medline]
    28. Amroze A, Field TS, Fouayzi H, Sundaresan D, Burns L, Garber L, et al. Use of Electronic Health Record Access and Audit Logs to Identify Physician Actions Following Noninterruptive Alert Opening: Descriptive Study. JMIR Med Inform 2019 Feb 07;7(1):e12650 [FREE Full text] [CrossRef] [Medline]
    29. Cash JJ. Alert fatigue. Am J Health Syst Pharm 2009 Dec 01;66(23):2098-2101. [CrossRef] [Medline]
    30. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-147 [FREE Full text] [CrossRef] [Medline]
    31. Ridgely M, Greenberg M. Too many alerts, too much liability: sorting through the malpractice implications of drug-drug interaction clinical decision support. J Health Law Policy 2012;5:257-296 [FREE Full text]
    32. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff (Millwood) 2011 Dec;30(12):2310-2317 [FREE Full text] [CrossRef] [Medline]
    33. Amante DJ, Hogan TP, Pagoto SL, English TM. A systematic review of electronic portal usage among patients with diabetes. Diabetes Technol Ther 2014 Nov;16(11):784-793. [CrossRef] [Medline]
    34. Irizarry T, DeVito DA, Curran CR. Patient Portals and Patient Engagement: A State of the Science Review. J Med Internet Res 2015;17(6):e148 [FREE Full text] [CrossRef] [Medline]
    35. HealthIT.gov. What is a patient portal?. 2014.   URL: https://www.healthit.gov/faq/what-patient-portal [accessed 2019-09-27]
    36. Britto MT, Hesse EA, Kamdar OJ, Munafo JK. Parents' perceptions of a patient portal for managing their child's chronic illness. J Pediatr 2013 Jul;163(1):280-1.e1. [CrossRef] [Medline]
    37. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015;17(2):e44 [FREE Full text] [CrossRef] [Medline]
    38. Kruse CS, Argueta DA, Lopez L, Nair A. Patient and provider attitudes toward the use of patient portals for the management of chronic disease: a systematic review. J Med Internet Res 2015;17(2):e40 [FREE Full text] [CrossRef] [Medline]
    39. Odone A, Ferrari A, Spagnoli F, Visciarelli S, Shefer A, Pasquarella C, et al. Effectiveness of interventions that apply new media to improve vaccine uptake and vaccine coverage. Hum Vaccin Immunother 2015;11(1):72-82 [FREE Full text] [CrossRef] [Medline]
    40. Jacobson VJC, Szilagyi P. Patient reminder and patient recall systems to improve immunization rates. Cochrane Database Syst Rev 2005(3):CD003941. [CrossRef] [Medline]
    41. Szilagyi PG, Albertin C, Casillas A, Valderrama R, Duru OK, Ong MK, et al. Effect of Patient Portal Reminders Sent by a Health Care System on Influenza Vaccination Rates: A Randomized Clinical Trial. JAMA Intern Med 2020 Jul 01;180(7):962-970. [CrossRef] [Medline]
    42. Orbell S, Hodgkins S, Sheeran P. Implementation Intentions and the Theory of Planned Behavior. Pers Soc Psychol Bull 1997 Sep;23(9):945-954. [CrossRef] [Medline]
    43. Pirolli P, Mohan S, Venkatakrishnan A, Nelson L, Silva M, Springer A. Implementation Intention and Reminder Effects on Behavior Change in a Mobile Health System: A Predictive Cognitive Model. J Med Internet Res 2017 Nov 30;19(11):e397. [CrossRef] [Medline]
    44. Thomas RE, Lorenzetti DL. Interventions to increase influenza vaccination rates of those 60 years and older in the community. Cochrane Database Syst Rev 2018 May 30;5:CD005188 [FREE Full text] [CrossRef] [Medline]
    45. Stockwell MS, Kharbanda EO, Martinez RA, Vargas CY, Vawdrey DK, Camargo S. Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial. JAMA 2012 Apr 25;307(16):1702-1708. [CrossRef] [Medline]
    46. Esteban-Vasallo MD, Domínguez-Berjón MF, García-Riolobos C, Zoni AC, Aréjula Torres JL, Sánchez-Perruca L, et al. Effect of mobile phone text messaging for improving the uptake of influenza vaccination in patients with rare diseases. Vaccine 2019 Aug 23;37(36):5257-5264. [CrossRef] [Medline]
    47. Regan AK, Bloomfield L, Peters I, Effler PV. Randomized Controlled Trial of Text Message Reminders for Increasing Influenza Vaccination. Ann Fam Med 2017 Nov;15(6):507-514 [FREE Full text] [CrossRef] [Medline]
    48. Krist AH, Woolf SH, Bello GA, Sabo RT, Longo DR, Kashiri P, et al. Engaging primary care patients to use a patient-centered personal health record. Ann Fam Med 2014;12(5):418-426 [FREE Full text] [CrossRef] [Medline]
    49. Lau AYS, Sintchenko V, Crimmins J, Magrabi F, Gallego B, Coiera E. Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial. J Am Med Inform Assoc 2012;19(5):719-727 [FREE Full text] [CrossRef] [Medline]
    50. Fischer GS, Hess R, Landeen BM, Weimer M, Zieth CR, Dong X, et al. Electronic reminders to patients within an interactive patient health record. Telemed J E Health 2013 Jun;19(6):497-500. [CrossRef] [Medline]
    51. Sequist TD, Zaslavsky AM, Colditz GA, Ayanian JZ. Electronic patient messages to promote colorectal cancer screening: a randomized controlled trial. Arch Intern Med 2011 Apr 11;171(7):636-641 [FREE Full text] [CrossRef] [Medline]
    52. Dharod A, Bellinger C, Foley K, Case LD, Miller D. The Reach and Feasibility of an Interactive Lung Cancer Screening Decision Aid Delivered by Patient Portal. Appl Clin Inform 2019 Jan;10(1):19-27. [CrossRef] [Medline]
    53. Kipping S, Stuckey MI, Hernandez A, Nguyen T, Riahi S. A Web-Based Patient Portal for Mental Health Care: Benefits Evaluation. J Med Internet Res 2016 Nov 16;18(11):e294 [FREE Full text] [CrossRef] [Medline]
    54. Sieck CJ, Hefner JL, McAlearney AS. Improving the patient experience through patient portals: Insights from experienced portal users. Patient Experience Journal 2018 Nov 06;5(3):47-54. [CrossRef]

    ‎
    CDC: Centers for Disease Control and Prevention
    OR: odds ratio

    Edited by G Eysenbach; submitted 21.10.19; peer-reviewed by C Kruse, A Scherer; comments to author 04.12.19; revised version received 24.06.20; accepted 03.08.20; published 25.09.20

    Copyright

    ©Jessica G Wijesundara, Mayuko Ito Fukunaga, Jessica Ogarek, Bruce Barton, Lloyd Fisher, Peggy Preusse, Devi Sundaresan, Lawrence Garber, Kathleen M Mazor, Sarah L Cutrona. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 25.09.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.