The overarching goal of this study was to improve rates of early influenza vaccination (by the end of October) among eligible adults in an outpatient population. Our primary objective was to determine whether our outreach increased completion of influenza vaccinations and, if so, whether one mode of outreach was most effective. Additional objectives were to improve documentation of influenza vaccinations administered outside the practice by inviting patient self-report (improving the accuracy of existing decision support tools) and to track process measures (eg, rates of portal message opening and interactive voice response call answering).

We conducted a nonblinded randomized controlled intervention (NCT02266277) at a large multispecialty medical group in central Massachusetts. Previously, our team developed and tested interactive voice response and portal outreach which we targeted to patients who had no documented vaccination 2 months after the start of the season [25,26]; this study adapted our previous approach, targeting a broader population in early September in order to promote early vaccination and provide information on September and October flu clinic dates.

Using a computer-generated randomization table, we assigned all active portal users to receive either a portal message promoting influenza vaccination, listing upcoming clinics, and offering online scheduling of vaccination appointments (n=19,506) or usual care with no portal message (n=19,505). Separately, we randomized all portal nonusers to receive either an interactive voice response call (n=15,000) or usual care with no interactive voice response call (n=43,596) (Figure 1). For portal users only, after the conclusion of the study and assessment of outcomes on November 3, messages were sent to the usual care group if they still did not have an electronic health record–documented influenza vaccination for the 2015-2016 season (n=14,118). The cost of calls prevented us from being able to send interactive voice response messages to the usual care group in the portal nonusers.

The study was reviewed and approved in 2014 by the Reliant Medical Group institutional review board; due to administrative changes, oversight was transferred to the University of Massachusetts institutional review board in 2015. A waiver for informed consent for patient outreach was approved by these institutional review boards. Patients were not compensated for their participation. The principal investigator (SC) oversaw the trial and data analysis.

This study consisted of interactive voice response or portal-based outreach. Qualitative interviews conducted with patients, physicians, nurses, and staff informed the development of our outreach material [25,26].

We designed an outgoing secure portal message to be sent via patient portal to patients randomized to the portal message arm (Multimedia Appendix 1). Portal message content appeared in letter format with the signature line reflecting the name of the patient’s primary care provider. Portal messages were delivered through standard channels used for portal-based correspondence between the medical group’s health care providers and patients (ie, a generic message that contained neither personal health information nor any reference to vaccinations was delivered to patient’s email account; the message prompted patients to log in to secure portal account via hyperlink). Once logged in to the portal accounts, patients clicked on a message labeled “A Message from Your Primary Care Provider” to view the outreach message. The outreach message included information about upcoming flu clinics for September and October 2015.

Unique to the portal message (compared to interactive voice response phone messages) was the option of direct online scheduling of appointments for influenza vaccination. Information about accessing the CDC vaccination website appeared within the body of the portal message as a hyperlink (and was conveyed verbally in the interactive voice response script). Opportunities to report external influenza vaccinations and to report intent to get vaccinated matched the interactive voice response call content.

Messages were sent out to 500 to 1500 patients daily over a period of 15 days in September 2015 in order to reduce the risk of being blocked by network bulk-spam filters.

Interactive voice response calls appeared on caller ID as originating from the medical group. This is consistent with identification of interactive voice response calls used for appointment reminders at the time of the study. Combining voice response with branching logic, calls elicited patient self-reports of influenza vaccinations completed outside the medical group (Multimedia Appendix 2). For patients reporting no influenza vaccination completed, calls included information about upcoming flu clinics for September and October 2015. Patients reporting no influenza vaccination completed were also asked whether they intended to get vaccinated. Patients reporting that they were unsure or did not intend to get vaccinated were asked further questions on specific reasons why they did not plan to get a flu vaccination.

Intervention interactive voice response calls, initiated on September 11, 2015, began by confirming that the person answering the phone was the intended patient recipient of the call. If voicemail was encountered or if the person reached was someone other than the patient, the interactive voice response system left a message asking patients to call back and provided an inbound call line number. The last interactive voice response outbound calls were placed on September 25, 2015. The inbound call line was maintained throughout the duration of outgoing calls and for 3 weeks after the final outgoing call was placed; patients who called this number from the phone number of record heard the interactive voice response call script in its entirety, beginning with questions confirming the identity of the caller.

With our proposed sample size, power calculations based on estimates of baseline vaccination rates indicated that 4286 participants per arm would give 80% power to detect a 3% improvement in influenza vaccination rates between groups (α=.05; 2-sided).

Baseline characteristics of both patient portal users and portal nonusers are reported in Table 1. Baseline characteristics were similar among portal users and portal nonusers. However, compared to the portal nonusers, portal users were more likely to be women, older, and have a higher level of health care utilization.

Among portal users, 28.4% (5549/19,506) of message recipients and 27.1% (5294/19,505) of the usual care group had documentation in their electronic health records that they received that influenza vaccinations on or before November 3, 2015 (P=.004). Portal users who received the messages were significantly more likely to have received the influenza vaccination compared to the usual care group (odds ratio [OR] 1.07, 95% CI 1.02-1.12). This finding was consistent even after adjusting for age, race, sex, and health care utilization (OR 1.07, 95% CI 1.02-1.12) (Table 2).

Among portal nonusers, 8.4% (1262/15,000) of call recipients and 8.2% (3586/43,596) of usual care recipients received vaccinations (P=.47). Bivariate and multivariate analysis showed nonsignificant differences in influenza vaccination rates between intervention and usual care groups (Table 2).

Among interactive voice response call recipients, 50.7% of patients were reached (7599/14,984); 36.8% (5513/14,984) responded to the first question asking if they received a flu vaccination on or after August 1, 2015, 3.8% (575/14,984) reported receiving their vaccination and 24.1% (3613/14,984) responded to a second question assessing their intention to receive a flu vaccination during the flu season (asked only of those who were not already vaccinated) (Table 4).

Of those reached, 72.5% (5513/7599) responded to our question on receipt of influenza vaccination, 7.6% (575/7599) reported already receiving a vaccination, and 47.5% (3613/7599) responded to our question on whether they planned to get vaccinated.

Our study showed a clinically small but statistically significant improvement in completion of early season influenza vaccinations among those randomized to receive outreach via patient portal, compared to a usual care control group (OR 1.07, 95% CI 1.02-1.12). There was no significant increase of early season influenza vaccinations among those randomized to receive an interactive voice response call (OR 1.03, 95% CI 0.96-1.10).

This outreach was designed to deliver a relatively time-sensitive message (eg, reminding patients of the importance of influenza vaccination and alerting them of upcoming vaccination clinics in September and October), and as such, represents a successful, brief patient engagement effort. We attained greater than 50% opening rates for portal messages and reached the targeted patient on over 50% of interactive voice response calls, achieving similar levels of exposure to the messaging for the two interventions. Among portal message recipients, more than one quarter of those who opened the message responded to our subsequent question on whether they had received or intended to receive their influenza vaccination. Among interactive voice response call recipients, close to three-quarters of those reached responded to the question.

A small number of patients reported influenza vaccinations that had been completed in the community; data on patient-reported vaccinations performed in the community were then uploaded into the electronic health record in order to improve the accuracy of existing influenza vaccination alerts directed at primary care providers. The relatively small number of self-reported flu vaccinations (961/20,978 participants across the portal message intervention and the interactive voice response intervention arms combined) has several possible explanations. It is possible that participants who had already been vaccinated found the outreach less relevant; and therefore, chose not to engage (eg, chose not to open the message or chose to hang up the phone). Relatively low rates of self-reported vaccination completion may also be attributable to the timing of our outreach—intentionally positioned at the start of the flu season. In contrast, our past work [26] captured much higher proportions of self-reported vaccinations (2591/20,000 intervention patients) when participants were approached several months past the start of flu season, which highlights the importance of asking patients to self-report vaccinations administered outside the clinic.

Although these “discovered” immunizations were incorporated into the electronic health record, they were not counted in the primary analysis; this allowed comparison with the usual care groups who were not queried for outside immunizations. While recognizing that the actual immunization rate was higher than the documented rate, any differences in the documented rate between intervention and control groups can reasonably be attributed to increased vaccination rates. Data on intention to be vaccinated were also elicited. Studies have shown that stating intent to complete a behavior can enhance likelihood of follow-through [42,43], thus it is possible that engaging patients in this manner may have contributed to the modest success of our intervention.

Once designed, portal messages (sent out in several batches) required minimal staff time to deliver. Our messages were designed to be easily adapted for use in future years; our messages have already been adapted and implemented by our medical center to target high-risk populations (eg, children with asthma).

There have been multiple studies [44-47] documenting success in use of patient-directed reminders for influenza vaccination. Effective reminders described include letters, telephone invitations, phone calls from a peer, tailored communications, customized letters/phone calls, and client-based appraisals [44]. Text message reminders have been shown to be successful for influenza vaccination reminders in pediatric and adolescent populations [45], in patients with rare diseases [46], and among high-risk patients [47].

While portal-based outreach directed at patients has been described for a variety of preventive measures [48], few studies have tested use of patient-directed influenza vaccination reminders sent via patient portals; one study [49] that has done so focused exclusively on untethered (ie, personally controlled) patient health records among a university student and staff population in Australia, yielding a 6.7% greater likelihood of influenza vaccination among users of the personal health records compared to those randomized to a 6-month waitlist for portal activation [49]. In our past work [25,26], we tested similar portal and interactive voice response outreach messages using a factorial design and targeting a population more likely to be nonadherent to vaccination guidelines (patients who had no documented vaccination 2 months after the start of the flu season). In that study [25,26], we found small but statistically significant improvements in influenza vaccination rates among recipients of either outreach method compared to usual care (portal message alone (OR 1.20, 95% CI 1.06-1.35); interactive voice response call alone (OR 1.15, 95 % CI 1.02-1.30); both messages (OR 1.29, 97.5% CI 1.13-1.48). In their recent randomized trial, Szilagyi et al [41] studied the effect of 1, 2, or 3 patient portal reminders on influenza vaccination rates among 164,205 patients in 52 primary care practices. They noted small but statistically significant improvements in vaccination rates with an attenuating effect of repeated messaging (37.5% for those receiving no reminders; 38.0% for 1 reminder; 38.2% for 2 reminders, and 38.2% for 3 reminders). Higher overall rates of vaccination were documented in their trial in comparison to ours; this is likely due in part to the study’s inclusion of children (who have a higher rate of influenza vaccination than that of the overall adult population) as well as their measurement of vaccination rates at the end of the influenza season (March 31).

The rates of patient engagement (measured by message opening and subsequent action) demonstrated by our patient portal recipients were comparable to those from previously published patient portal outreach studies. In our previous influenza outreach intervention [25,26] (in a population unvaccinated 2 months into the flu season), messages were opened by slightly more than half of message recipients and interactive voice response targets were reached in just over 60%. In that prior study, among those who opened portal messages, 28.6% responded to subsequent questions. Among those reached via interactive voice response call, 78.3% responded to questions. In their patient portal intervention aimed at improving influenza vaccination rates, Szilagyi and colleagues [41] found opening rates of 52.9% among patients receiving a single portal message (increasing to 55.9% in a 2-reminder group and 58.8% in a 3-reminder group).

Fischer and colleagues [50] found that among patients randomized to active reminders about multiple health maintenance services (eg, hemoglobin A_1c testing, lipids, etc), nearly 65% of patients logged in to the portal after receiving the first of several messages. In a study [51] of colorectal cancer screening reminders delivered via patient portal, among the 552 patients randomized to receive messages, 54% viewed the message and 9% performed a suggested web-based risk assessment tool. In a study describing the reach and feasibility of an interactive lung cancer screening decision aid delivered by patient portal, Dharod and colleagues [52] found that 86% of lung cancer screening eligible patients identified by an electronic health record algorithm to receive a patient portal message read the message, 40% then visited a web-based decision aid for lung cancer screening, and 35% completed questionnaires to determine their eligibility for lung cancer screening.

Our study had several limitations. Despite responding to inquiries on flu vaccinations receipt at higher rates, the interactive voice response group did not show a significant impact on vaccination rates (while the portal message group showed a small statistically significant improvement in vaccination rates). This discrepancy could be due in part to differences in the baseline characteristics of portal nonusers compared to portal users. For several reasons, interactive voice response call recipients may have been more resistant to completion of influenza vaccination than portal message recipients. Portal nonusers were less likely to have had an office visit in the previous 12 months and might have been less actively engaged in their health care overall. In addition, eligibility for the interactive voice response study was intentionally more stringent, with patients included only if they lacked documentation of an influenza vaccination in the previous flu season. This study design choice was made with guidance from the medical center where our study was implemented and was intended to make available the limited resource of interactive voice response calls to the broadest possible number of nonadherent patients (our study covered the cost for only 15,000 calls but we had no limit on the number of portal messages that we could send). It is possible that the interactive voice response calls might have yielded a greater impact if we had not opted to employ this more stringent eligibility criterion.

There are compelling reasons to use existing functionality within electronic health record–tethered portals to promote influenza vaccination. For vaccination outreach, data recorded in the electronic health record through routine care delivery can inform real-time identification of unvaccinated populations. Portal-based outreach can be more cost-effective than phone calls or mailings and easier to implement than a new app; simple messages can be sent out by office staff without additional informatics expertise. Studies [37,38,53,54] show that patient portals can enhance patient empowerment and sense of autonomy, enhance patient engagement, improve medication adherence, decrease office visits, increase self-management of disease and disease awareness, increase use of preventive medicine, and increase inclusion of patients in medical decision making.

Our parallel interventions to patient portal users and portal nonusers allowed us to assess the impact of outreach supporting influenza vaccination in both groups. Our study demonstrated a small improvement in influenza vaccination rates among portal message recipients, and successful patient engagement in both portal message recipients and interactive voice response call recipients. This method is readily applicable to current practice.

As we simultaneously face the upcoming influenza season and the ongoing COVID-19 pandemic, expanding influenza vaccination coverage in ambulatory populations can decrease the strain on our overtaxed health system and may help avert poor outcomes in patients at risk for co-infection with influenza and COVID-19.

Our findings of a small but significant improvement in influenza vaccination rates resulting from portal-based outreach represent an important contribution to the national conversation on caring for and protecting our patients in the upcoming months and for years to come.