Published on in Vol 22, No 2 (2020): February

Preprints (earlier versions) of this paper are available at, first published .
Characteristics of Patients Using Different Patient Portal Functions and the Impact on Primary Care Service Utilization and Appointment Adherence: Retrospective Observational Study

Characteristics of Patients Using Different Patient Portal Functions and the Impact on Primary Care Service Utilization and Appointment Adherence: Retrospective Observational Study

Characteristics of Patients Using Different Patient Portal Functions and the Impact on Primary Care Service Utilization and Appointment Adherence: Retrospective Observational Study

Original Paper

1Department of Industrial and Systems Engineering, University of Florida, Gainesville, FL, United States

2Public Health Sciences Division, Fred Hutchinson Cancer Research Center, Seattle, WA, United States

3University of Florida Health Physicians, Gainesville, FL, United States

Corresponding Author:

Xiang Zhong, PhD

Department of Industrial and Systems Engineering

University of Florida

482 Weil Hall

Gainesville, FL, 32611

United States

Phone: 1 3522947716


Background: Patient portals are now widely available and increasingly adopted by patients and providers. Despite the growing research interest in patient portal adoption, there is a lack of follow-up studies describing the following: whether patients use portals actively; how frequently they use distinct portal functions; and, consequently, what the effects of using them are, the understanding of which is paramount to maximizing the potential of patient portals to enhance care delivery.

Objective: To investigate the characteristics of primary care patients using different patient portal functions and the impact of various portal usage behaviors on patients’ primary care service utilization and appointment adherence.

Methods: A retrospective, observational study using a large dataset of 46,544 primary care patients from University of Florida Health was conducted. Patient portal users were defined as patients who adopted a portal, and adoption was defined as the status that a portal account was opened and kept activated during the study period. Then, users were further classified into different user subgroups based on their portal usage of messaging, laboratory, appointment, and medication functions. The intervention outcomes were the rates of primary care office visits categorized as arrived, telephone encounters, cancellations, and no-shows per quarter as the measures of primary care service utilization and appointment adherence. Generalized linear models with a panel difference-in-differences study design were then developed to estimate the rate ratios between the users and the matched nonusers of the four measurements with an observational window of up to 10 quarters after portal adoption.

Results: Interestingly, a high propensity to adopt patient portals does not necessarily imply more frequent use of portals. In particular, the number of active health problems one had was significantly negatively associated with portal adoption (odds ratios [ORs] 0.57-0.86, 95% CIs 0.51-0.94, all P<.001) but was positively associated with portal usage (ORs 1.37-1.76, 95% CIs 1.11-2.22, all P≤.01). The same was true for being enrolled in Medicare for portal adoption (OR 0.47, 95% CI 0.41-0.54, P<.001) and message usage (OR 1.44, 95% CI 1.03-2.03, P=.04). On the impact of portal usage, the effects were time-dependent and specific to the user subgroup. The most salient change was the improvement in appointment adherence, and patients who used messaging and laboratory functions more often exhibited a larger reduction in no-shows compared to other user subgroups.

Conclusions: Patients differ in their portal adoption and usage behaviors, and the portal usage effects are heterogeneous and dynamic. However, there exists a lack of match in the patient portal market where patients who benefit the most from patient portals are not active portal adopters. Our findings suggest that health care delivery planners and administrators should remove the barriers of adoption for the portal beneficiaries; in addition, they should incorporate the impact of portal usage into care coordination and workflow design, ultimately aligning patients’ and providers’ needs and functionalities to effectively deliver patient-centric care.

J Med Internet Res 2020;22(2):e14410



The patient-centric care initiative heightened the awareness of health care systems’ responsibility to provide easily accessible ways for patients to engage in their own care and become effective health care partners. Such a mission is expected to be fulfilled with patient portals, where a portal is defined as “a secure online website that allows patients to access their medical records or communicate with their health care providers” [1]. Empowered by the rapid development of health information technologies, patient portals are now widely available and increasingly adopted by patients and providers. Effective use of these portals is expected to result in improved care access, self-management, and care coordination. Furthermore, the US federal government has authorized incentive payments to physicians who demonstrated “meaningful use” of such health information systems [2]. Consequently, patient portal research has garnered growing attention; a spate of reports of portal adoption and enrollee demographics have been published over the past decade. These studies typically described individual portal deployment or analyzed national survey data, such as the Health Information National Trends Survey (HINTS), and they reported characteristics of early portal adopters [3-7]. Along with this, whether the adoption of patient portals affects health care consumption was also investigated. Health care consumption (ie, the usage of various clinical services) is closely related to care access and coordination and is, thus, an important decision factor for service operations. Understanding the impact of portals on health care consumption can facilitate the design of service systems that accommodate patients’ portal usage, leading to enhanced efficiency of service operations and improved patient access to care. However, most reviews reported mixed evidence about the effect of patient portals on health care consumption—whether portal adoption will increase or decrease outpatient office visits was debated [8-12]—and the only consensus was that patient portals were used as a complement rather than a substitute of usual clinical services [13-16]. In addition, the number of appointment no-shows has been chosen to serve as an indicator to infer patient engagement [17], and it has been reported that portal enrollment is significantly related to decreases in appointment no-shows [18-20]. However, such studies mainly captured the association but not the causation between portal enrollment and no-show reduction. It motivated us to carry out a study that could account for measurable confounders and is robust to unmeasured confounders, hence, unveiling the causal effect of portal usage. In particular, we chose to investigate primary care office visit and telephone encounter rates (per quarter) as the measures of primary care service utilization as well as appointment cancellation and no-show rates (per quarter) as the measures of appointment adherence.

Despite the growing interest in portal adoption, there is a notable paucity of follow-up studies describing whether patients use portals actively and how distinct portal functions, such as messaging, laboratory, appointment, and medication, are used after adoption. The successful achievement of the promise to improve care access, self-management, and care coordination is intrinsically linked to the extent to which portals are used. We hypothesized that a patient who actively communicates with physicians using secure messages will benefit more from adopting a patient portal than one who never uses it after adoption. This is evidenced by the literature that states that messaging usage is associated with patient engagement [21,22]. In addition to messaging, the appointment function of portals offers an alternative way to make appointments than by phone calls and makes it easier for patients to reschedule or cancel their appointments. It can be hypothesized that with more freedom to manage appointments, patients will be more adherent to their appointments. Furthermore, with the laboratory and the medication functions of portals, patients can easily access their lab results and refill prescriptions online. Reminders can be sent from portals as an intervention to encourage patients to check their test results or refill their medications. Overall, we believed the convenience brought by patient portals to patients will enable them to be the owners of their health and be actively engaged in their care management. Lastly, we hypothesized that patients’ portal usage behaviors are heterogeneous, and different portal functions might be perceived with distinct values by users with various characteristics. To test these hypotheses, it is necessary to look at how patient characteristics are associated with portal usage and how different portal usage patterns affect patients’ care consumption and adherence to appointments.

The evaluation of how often patients access portals and what they do with them was not given enough attention in the past. Ignoring such a variety behaviors of patients might lead to the misspecification of portal effects on patients. For instance, if a subgroup of patients is doing significantly better with portals, whereas another group is doing worse, aggregating them can potentially lead to a conclusion of “no change.” Furthermore, portal adoption and the subsequent usage can influence patients over time, and the time trend in portal usage effects should not be overlooked; from an operational and a strategic point of view, both short-term and long-term impacts matter. Therefore, instead of solely relying on the observations from cross-sectional data, we sought to examine longitudinal data and focused on not only portal adoption but also on portal usage, aiming to investigate the following: (1) the characteristics of people who are more likely to adopt a patient portal, (2) among patients who have adopted portals, determine who uses portals more often and the characteristics of people associated with different portal function usage behaviors, and (3) whether the primary care service utilization and appointment adherence of patients who have adopted portals are affected by their different portal usage behaviors, featuring both the amount of use and the type of portal functions used. The answers to these questions are vital to informing (1) the design and implementation of patient portals, (2) the service system operations, such as the daily workflow design, and (3) the policy guidelines, such as remuneration models to compensate providers’ portal time.

Study Setting

Data Source

This study used the data generated by a large primary care patient cohort affiliated with the University of Florida (UF) Health; the data protocol was approved by the UF Institutional Review Board. In 2011, UF Health started offering an electronic patient portal named MyUFHealth, or MyChart, which allows patients to access to portions of their medical records (eg, released test results and after-visit summaries), communicate with the clinical service providers using secure messaging, request prescription refills, and manage outpatient appointments. Monthly clinical service utilization and portal activities of individual patients were generally not frequent; therefore, the time unit used in this study was one quarter: January-March (first quarter, Q1), April-June (second quarter, Q2), July-September (third quarter, Q3), and October-December (fourth quarter, Q4). For instance, Y13 Q3 stands for the third quarter of the year 2013. The study period was from July 1, 2013, to June 30, 2016. During the study period, there were 46,544 UF Health patients who had at least one visit to UF Health family medicine clinics. More than 95% of them came from North Central Florida.

Study Sample

Because the portal accounts of patients under 18 years old are typically managed by their legal guardians, we restricted our analysis to adult patients. We further restricted the study to insured patients who (1) chose UF Health as their primary health care provider, (2) enrolled in UF Health before the start of the study period, and (3) maintained an enrollment status until the end of the study period. As such, their primary care service utilization within the UF Health network can be fully captured. It is worth noting that UF Health is the leading care provider in the study region, and primary care services rendered to insured patients outside of the UF Health network are very limited. In addition, to ensure a contrast of before-after portal adoption and to capture the portal usage effects over time, we defined users in our study as patients who adopted MyUFHealth during periods Y13 Q4 to Y15 Q3. We excluded patients who (1) adopted the portal before the study period, (2) adopted the portal relatively recently (ie, adopted the portal in Y15 Q4 or after), or (3) were temporary users (ie, adopted the portal but closed their accounts before the study ended). These inclusion and exclusion criteria led to 17,580 nonusers and 4312 users.

Variables and Measures

Patients’ demographic and socioeconomic information, including age category, gender, race or ethnicity, marital status, insurance type, and their active problem number (APN), were obtained from their electronic medical records (EMRs). The APN is the number of problems in a patient’s active problem list, which captures patients’ chronic conditions and any ongoing impactful conditions that are resolvable but are important for physicians to be aware of to make clinical decisions. Notably, an ailment like a common cold or flu does not appear in the active problem list, and this list is typically reviewed at each patient encounter and updated—adding or deleting problems—whenever deemed necessary. Accordingly, a patient’s APN is considered as a time-varying confounder to account for individual disease burdens. It should also be noted that patients tend to use care services intensively right after an onset diagnosis of a new health condition and less frequently later, due to the resolution of the triggering health care condition [8]. A patient’s disease process (ie, an onset of a condition, followed by an episode of treatment, possibly including the resolution of the condition) can be nested within the process of portal adoption and subsequent usage. Therefore, we proposed a study design that controls for the time a new diagnosis was made (ie, when a visit type coded as new appeared in the EMR), allowing an assessment of the natural disease process.

To characterize portal usage patterns, we focused on four major portal functions that are regularly accessed by users: messaging (MESG), laboratory (LAB), medication (MED), and appointment (APPT). Patient portal usage is measured by the amount of use per quarter by function type. In particular, variable MESGit is defined as the count of messaging-related activities by user i at quarter t, such as open a message box, read a message, delete a message, and send a message by patients. Variables LABit, MEDit, and APPTit represent the count of actions related to laboratory activities (eg, check lab test results and request lab test); actions related to medication, such as check medication list and request drug or prescription refill; as well as actions related to appointments, such as appointment scheduling, appointment status checking, and cancel or reschedule an appointment, respectively.

To evaluate how portal usage affects primary care service utilization and appointment adherence, rates of office visits categorized as arrived, cancelled, or no-show, as well as telephone encounters per quarter were measured. Patients’ office visits and telephone encounters within the UF Health network were used as an indicator of their overall primary care service utilization.

User Subgroup Clustering

Patients’ portal activities differ across individuals and time: they might use a specific portal function more or less frequently based on their intrinsic preferences or immediate care needs, which might change with time. Therefore, we considered the portal usage over the course of a postadoption phase as the exposure and the use of primary care services as the outcome. We aimed to investigate the causality by examining the time dynamic behaviors in both exposures and outcomes. To characterize the time-varying exposures, we categorized patients into user subgroups and investigated the makeup of each subgroup, as well as the portal activity features associated with each subgroup.

Specifically, to cluster patients, we defined an activity feature vector (MESGit,LABit,MEDit,APPTit) (ie, the amount of function usage by user i at t quarters postadoption). This entails a pattern recognition problem with each user i being characterized by a set of ordered vectors , where Ti is the set of observation times postadoption for user i. A naive treatment is to take the average utilization over time and create a compact feature vector. However, this cannot separate the cases where a patient was moderate in messaging utilization in each quarter, versus a patient who did not use messaging except for one quarter of intensive use that brings the average utilization into the moderate level. Therefore, we proposed a two-stage clustering method to mitigate the flaw of averages and allow some assessment of the longitudinal usage patterns.

In the first stage, we characterized the relationships between four functions, for instance, whether there were two or more functions that were frequently used together at any time. The spherical clustering method [23] was employed to cluster activity feature vectors (MESGit,LABit,MEDit,APPTit). The difference in scales can be addressed by this method. For instance, the overall usage of messaging is one order of magnitude higher than that of medication. As a result, five activity clusters—CMESG, CLAB, CMED, CAPPT, and CM&L—were identified, which were named after their dominant activities. For instance, if patient i used messaging many times but not so much for the rest of the functions at time t, the activity feature vector will then be labeled with “CMESG” at time t. For activity feature vectors with MESG and LAB functions used together and more often than others, a label of “CM&L” was assigned. In addition, a sixth cluster named CSilent was assigned for any activity feature vector being a vector of zeros. After the label assignment, a patient was then characterized by a |Ti|-dimensional pattern vector, (Ci1, Ci2,..., Ci|Ti| ), where . For example (CMESG, CSilent, CMESG, CSilent, CSilent) is a labeled pattern vector for a patient with five observations postadoption (ie, Ti=5). Based on analyzing the data, the order of labels was quite random and, thus, was not featured into user types.

In the second stage, each user was assigned to one user type based on the number of occurrences of various activity clusters (ie, labels) over the postadoption period. For instance, with the above sample patient, cluster CMESG has a frequency of 2/5, cluster CSilent has a frequency of 3/5, and the frequency is 0 for the rest of the clusters. A user feature vector (2/5,0,0,0,0,3/5) representing the frequencies of belonging to clusters {CMESG, CLAB, CMED, CAPPT, CM&L, CSilent} was created for the patient-level clustering. Because the user feature vector was already normalized, the K-means with Euclidian distance clustering method was used. As a result, five clusters were identified to represent five user types—UMESG, ULAB, UAPPT, UM&L, and USilent—named after their dominant activity clusters. For instance, USilent represents the type of users who were, in general, inactive postadoption.

To summarize, we created activity feature vectors and collected activity feature vectors of all patients at every quarter after portal adoption to find common patterns, referred to here as activity clusters. This concluded the first-stage clustering. We then labeled each activity feature vector—per patient per time—with a membership (ie, belonging to one out of six activity clusters). This yielded a longitudinal activity pattern vector for each patient. Notably, such pattern vectors were of different dimensions, due to different observational time lengths postadoption. Thus, they were further mapped to a user feature vector with a fixed dimension of six for each patient. The user feature vectors of all patients were then clustered, which led to the final five user subgroups. This concluded the second-stage clustering. The illustration of the two-stage clustering and the descriptive statistics of the clustering results can be found in Multimedia Appendix 1.

Vector Matching Using Propensity Scores

Confounders are the determinants of exposure that are associated with outcomes (ie, variables that potentially affect both outcomes), for example, primary care service utilization, and exposure to different types of interventions (ie, becoming a specific type of user). To reduce the bias in causal inference due to confounders, we matched the users and nonusers using the vector matching method [24]. We first calculated propensity scores by estimating the probability of belonging to a user subgroup using multinomial logit regression (MLR). The covariates of the MLR model include the time-invariant characteristics (ie, age category, gender, race, marital status, and insurance type) and the time-varying variables (ie, APN and primary care office visits categorized as arrived, no-show, or cancellation, as well as telephone encounters) measured at their baseline. Notably, only the most recent marital status was recorded in the system. In addition, insurance types can change over time; however, the change was infrequent in our patient population, due to a relatively limited study time span. Therefore, a patient’s insurance type was treated as a time-invariant variable. The baseline values are observations averaged over the time period before adoption for users, and before Y14 Q4 for nonusers. The matching was based on the propensity score vectors obtained from MLR. A nonuser was matched to a user in one subgroup with a similar propensity score vector. To enhance the matching outcome, we further allowed nonusers to be exactly matched to users upon having multiple candidates available in the same propensity score stratum.

Generalized Linear Model for Heterogeneous Portal Usage Effects

In addition to matching patient demographics, the time-varying disease burden and the dynamic disease process needed to be addressed, which motivated a causal inference study accounting for both time-invariant and time-varying confounders. A panel difference-in-differences (DID) framework using generalized linear models was developed. The framework was similar to that in Zhong et al [25] but was generalized to capture heterogeneous portal usage effects. The detail of the model can be found in Multimedia Appendix 1.

Using such a framework, one can estimate the rate ratios (RRs) between the users and the matched nonusers for the targeted outcomes, including rates (per quarter) of office visits categorized as arrived, cancelled, or no-show, as well as telephone encounters. RRs for different user subgroups after portal adoption with an observational window of up to 10 quarters were obtained. An RR being significantly greater than 1 at a given quarter postadoption implies that the corresponding rate (eg, office visit rate) of the users was significantly larger than that of the nonusers at that quarter, which measures the time-dependent portal effects. In this study, all statistical analyses were performed using R, version 3.3.1 (The R Foundation), with two-sided statistical tests at a .05 significance level.

Patient Portal Adoption

A logistic regression model was built to predict portal adoption; the odds ratios (ORs) obtained are exhibited in Table 1. The following were negatively associated with portal adoption: Hispanic and black or African American race versus white (OR 0.38 vs 0.53, 95% CI 0.19-0.69 vs 0.69-0.92, P=.003 vs P<.001); male gender (OR 0.64, 95% CI 0.59-0.68, P<.001); marital status as not married (ie, single, divorced, widowed, or not having a life partner or a significant other) in contrast to married (ORs 0.30-0.72, 95% CIs 0.22-0.83, all P<.001); and insurance type as not being Blue Cross Blue Shield (ORs 0.23-0.73, 95% CIs 0.12-0.80, all P<.001). Moreover, a high baseline APN (ORs 0.57 and 0.86, 95% CIs 0.51-0.63 and 0.80-0.94, all P<.001) and a high baseline no-show rate (OR 0.29, 95% CI 0.21-0.40, P<.001) were negatively associated with portal adoption. The following were positively associated with portal adoption: being above 30 years of age in contrast to being 19-30 years of age (ORs 1.20-1.28, 95% CIs 1.07-1.52, all P<.01) and having a high baseline telephone encounter rate (OR 1.13, 95% CI 1.06-1.19, P<.001).

Patient Portal Usage

Portal users’ usage summary statistics are presented here. The mean portal log-in rate was 6.85 per user per quarter (SD 12.11) with a median of 3 (IQR 8). The most frequently used portal function (per quarter) was messaging (mean 17.67, SD 34.69; median 5, IQR 20), followed by laboratory (mean 12.22, SD 28.04; median 1, IQR 13), appointment (mean 7.65, SD 19.89; median 1, IQR 7), and medication (mean 1.73, SD 4.05; median 0, IQR 2). The average number of secure messages sent from patients—we counted unique conversation threads, which can include multiple back-and-forth messages—was 1.07 per quarter. It was observed that 1214 users were very active and constantly accessed the portal postadoption. The remaining users did not use the portal in at least one quarter after adopting it.

Table 1. Odds ratios (ORs) of patient characteristics for portal adoption.
Patient characteristicsOR (95% CI) (users versus nonusers)P value
Age in years (reference: 19-30)

31-451.22 (1.09-1.36)<.001

46-641.20 (1.07-1.34).002

65+1.28 (1.07-1.52).007
Gender (reference: female)

Male0.64 (0.59-0.68)<.001
Race (reference: white)

Asian1.17 (0.96-1.42).11

Black or African American0.53 (0.48-0.58)<.001

Hispanic0.38 (0.19-0.69).003

Others0.80 (0.69-0.92).002
Marital status (reference: married or companion)

Divorced or separated0.72 (0.61-0.83)<.001

Other0.30 (0.22-0.39)<.001

Single0.66 (0.60-0.71)<.001

Widowed0.50 (0.40-0.62)<.001
Insurance type (reference: Blue Cross Blue Shielda)

Commercial or managed care0.73 (0.66-0.80)<.001

Medicaid0.42 (0.36-0.48)<.001

Medicare0.47 (0.41-0.54)<.001

Other0.23 (0.12-0.40)<.001

Self-pay0.46 (0.39-0.55)<.001
Baseline care service utilization (continuous)

Telephone encounter1.13 (1.06-1.19)<.001
Office visit: arrived0.97 (0.90-1.04).41
Office visit: cancelled1.06 (0.94-1.19).36
Office visit: no-show0.29 (0.21-0.40)<.001
Baseline APNb (reference: ≤2.5)

>2.5 and ≤70.86 (0.80-0.94)<.001

>70.57 (0.51-0.63)<.001

aBlue Cross Blue Shield is a type of commercial insurance with a sufficiently large body of enrollees that can enable us to statistically identify its effect.

bAPN: active problem number.

Patient User Subgroups

After the two-stage clustering, we identified 615, 663, 1006, 536, and 1492 patients in user subgroups, ULAB (14.3%), UM&L (15.4%), UMESG (23.3%), UAPPT (12.4%), and USilent (34.6%), respectively. The association between user types and patient characteristics was analyzed using MLR. Patients’ baseline care service utilization and marital status were not significantly associated with user types (P>.05). Married people or people with a life partner or a significant other, although being more likely to adopt portals compared to single people, were not less likely to be USilent. The ORs of the significantly relevant patient characteristics obtained from the MLR model are shown in Table 2. It can be seen that age is the most important predictor of user types. The ORs of using appointment functions strictly decrease with age (31-45, 46-64, and 65+ years: ORs 0.52, 0.34, and 0.19, respectively, 95% CIs 0.11-0.67, all P<.001). On the contrary, the intention of using messaging increases with age (31-45 years: OR 0.99, 95% CI 0.76-1.29, P>.05; 46-64 and 65+ years: ORs 1.38 and 1.50, 95% CIs 1.07-1.78 and 1.00-2.23, P=.01 and .04, respectively). Regarding gender, males used the laboratory function less often, such as being type UM&L (OR 0.61, 95% CI 0.50-0.76, P<.001) and being type ULAB (OR 0.75, 95% CI 0.61-0.93, P=.01) in contrast to being silent. In addition, male users were more inactive: no ORs were significantly larger than 1. On race and ethnicity, compared to white users, Asian users used the messaging less (OR 0.44, 95% CI 0.26-0.73, P=.002) but used the laboratory more (OR 1.58, 95% CI 1.04-2.39, P=.03). Black or African American users made significantly more appointments via the portal (OR 1.36, 95% CI 1.05-1.76, P=.02). Hispanic users were relatively silent: ORs were insignificant due to a small sample size.

On insurance type, it is interesting to note that although Medicaid and Medicare patients tended not to adopt a portal compared to Blue Cross Blue Shield patients, Medicaid patients shared a similar user type distribution as Blue Cross Blue Shield patients (for all ORs, P>.05). Moreover, Medicare patients used messaging significantly more (OR 1.44, 95% CI 1.03-2.03, P=.04) and were less inactive compared to other insurance types: no ORs were significantly smaller than 1. It should be noted that Medicare patients also include those who are less than 65 years of age but have received Social Security Disability Insurance checks for at least 24 months or have been diagnosed with end-stage renal disease [26]. In our patient population, we have around 28% of Medicare patients who were less than 65 years of age.

Lastly, a heavy disease burden in contrast to a small APN was significantly positively associated with frequent portal usage of any activity types (ORs 1.37-1.76, 95% CIs 1.11-2.22, all P≤.01), which is contrary to the observation that patients with a heavy disease burden tended not to adopt a portal. To demonstrate the quality of matching, the characteristics of the users and the nonusers before and after matching are shown in Table 3.

Table 2. Odds ratios (ORs) of patient characteristics for being in different user subgroups.
Patient characteristicsNonsilent versus silent users
User typesLABa,
OR (95% CI)
P value MESGb and LAB,
OR (95% CI)
P value MESG,
OR (95% CI)
P value APPTc,
OR (95% CI)
P value
Age categories (years) (reference: 19-30)

31-450.76 (0.58-1.00).050.85 (0.64-1.13).250.99 (0.76-1.29).920.52 (0.40-0.67)<.001

46-640.83 (0.63-1.09).181.11 (0.84-1.47).451.38 (1.07-1.78).010.34 (0.25-0.45)<.001

65+0.77 (0.48-1.25).291.01 (0.64-1.59).981.50 (1.01-2.23).040.19 (0.11-0.33)<.001
Gender (reference: female)

Male0.75 (0.61-0.93).010.61 (0.50-0.76)<.0010.95 (0.80-1.13).571.13 (0.91-1.41).26
Race (reference: white)

Asian1.58 (1.04-2.39).030.86 (0.54-1.39).550.44 (0.26-0.73).0020.54 (0.29-1.00).05

Black or African American1.18 (0.91-1.53).220.84 (0.64-1.10).200.90 (0.71-1.14).381.36 (1.05-1.76).02

Hispanic0.75 (0.15-3.66).720.36 (0.04-2.99).350.29 (0.04-2.36).25N/Ad.94

Other1.29 (0.90-1.86).170.90 (0.61-1.33).600.89 (0.63-1.25).490.95 (0.63-1.42).79
Insurance (reference: Blue Cross Blue Shield)

Commercial or managed care0.82 (0.64-1.06).130.70 (0.54-0.90).011.04 (0.84-1.29).710.62 (0.47-0.82)<.001

Medicaid1.09 (0.74-1.60).651.00 (0.68-1.47).991.10 (0.77-1.57).590.88 (0.60-1.31).54

Medicare1.27 (0.83-1.94).271.25 (0.84-1.87).281.44 (1.03-2.03).041.43 (0.90-2.29).13

Other0.35 (0.04-2.86).330.54 (0.11-2.63).450.20 (0.02-1.68).140.45 (0.05-3.70).45

Self-pay0.53 (0.30-0.94).030.56 (0.33-0.96).031.00 (0.66-1.51)>.990.87 (0.53-1.43).58
Baseline APNe(reference: ≤2.5)

>2.5 and ≤71.56 (1.26-1.94)<.0011.37 (1.11-1.70).0041.37 (1.13-1.65).0011.45 (1.15-1.82).001

>71.52 (1.14-2.01).0041.66 (1.27-2.16)<.0011.76 (1.40-2.22)<.0011.53 (1.13-2.08).01

aLAB: laboratory.

bMESG: messaging.

cAPPT: appointment.

dNot applicable—the CI cannot be estimated due to the small sample size.

eAPN: active problem number.

Table 3. Patient characteristics of unmatched nonusers, users, and matched nonusers.
CharacteristicsUnmatched nonusers (N=17,580), n (%)Users (N=4024), n (%)P value Matched nonusers (N=4024), n (%)P value
Age categories (years)


19-303777 (21.48)769 (19.11)
803 (19.96)

31-454220 (24.08)1152 (28.63)
1181 (29.35)

46-645662 (32.21)1418 (35.24)
1364 (33.90)

65+3921 (22.30)685 (17.02)
676 (16.80)


Female10,443 (59.40)2648 (65.81)
2610 (64.86)

Male7137 (40.60)1376 (34.19)
1414 (35.14)


Asian350 (1.99)119 (2.96)
119 (2.96)

Black or African American5509 (31.34)675 (16.77)
675 (16.77)

Hispanic91 (0.52)9 (0.22)
9 (0.22)

Other1035 (5.89)241 (5.99)
241 (5.99)

White10,595 (60.27)2980 (74.06)
2980 (74.06)
Marital status


Divorced or separated1372 (7.80)222 (5.52)
231 (5.74)

Married or companion6820 (38.79)2197 (54.60)
2196 (54.57)

Other472 (2.68)53 (1.32)
41 (1.02)

Single8054 (45.81)1461 (36.31)
1477 (36.70)

Widowed862 (4.90)91 (2.26)
79 (1.96)


Blue Cross Blue Shield5986 (34.05)2122 (52.73)
2114 (52.53)

Commercial or managed care2797 (15.91)742 (18.44)
755 (18.76)

Medicaid2683 (15.26)278 (6.91)
257 (6.39)

Medicare4906 (27.91)718 (17.84)
726 (18.04)

Other145 (0.82)11 (0.27)
17 (0.42)

Self-pay1063 (6.05)153 (3.80)
155 (3.85)
Baseline APNa


≥0 and ≤2.55523 (31.42)1704 (42.35)
1704 (42.35)

>2.5 and ≤76090 (34.64)1482 (36.83)
1509 (37.50)

>75967 (33.94)838 (20.83)
811 (20.15)

aAPN: active problem number.

Primary Care Service Utilization and Appointment Adherence


Using the panel DID models (see Multimedia Appendix 1), we compared the utilization of primary care services between the matched nonusers and different user subgroups before and after portal adoption. The RRs of the users to the nonusers attributable to portal usage effects at each quarter postadoption are shown in Figure 1, and the corresponding RRs can be found in Table 4. The RRs measure the time-varying difference between the portal users and the matched nonusers at each quarter after portal adoption. To interpret, an RR being 1 suggests that there is no difference between the users and the matched nonusers, and thus there is no significant portal effect.

Figure 1. Quarterly rate ratios (users/nonusers) of primary care office visits categorized as arrived, cancellation, no-show, as well as telephone encounters postadoption of the portal. APPT: appointment; LAB: laboratory; MESG: messaging; M&L: messaging and laboratory; Silent: being inactive.
View this figure
Table 4. Quarterly rate ratios (RRs) between the portal users and the matched nonusers of office visits categorized as arrived, telephone encounter, cancellation, and no-show for different user subgroups after portal adoption.
User type and periodaArrivedTelephone encounterCancellationNo-show

RR (95% CI)P value RR (95% CI)P value RR (95% CI)P value RR (95% CI)P value

P01.91 (1.72-2.11)<.0012.07 (1.78-2.37)<.0010.89 (0.71-1.08).270.60 (0.36-0.83).001

P11.36 (1.20-1.52)<.0011.29 (1.06-1.51).0041.04 (0.80-1.27).780.71 (0.42-1.01).06

P21.17 (1.02-1.32).021.28 (1.06-1.50).011.05 (0.79-1.31).700.67 (0.37-0.97).03

P31.23 (1.07-1.39).0021.36 (1.13-1.60)<.0010.99 (0.74-1.23).910.71 (0.40-1.02).07

P41.27 (1.10-1.44)<.0011.21 (0.97-1.44).060.75 (0.54-0.97).030.69 (0.37-1.00).05

P51.25 (1.07-1.43).0031.25 (0.99-1.50).040.92 (0.65-1.19).580.49 (0.21-0.78).001

P61.08 (0.89-1.27).391.09 (0.83-1.35).491.02 (0.69-1.36).890.55 (0.19-0.92).02

P70.99 (0.80-1.18).930.96 (0.70-1.22).771.06 (0.68-1.44).760.90 (0.36-1.44).72

P81.00 (0.78-1.22)>.990.93 (0.63-1.23).671.07 (0.62-1.52).770.75 (0.17-1.33).40

P91.02 (0.76-1.28).881.09 (0.72-1.45).630.97 (0.50-1.44).900.96 (0.21-1.70).91

P101.00 (0.69-1.31).981.06 (0.60-1.52).791.15 (0.48-1.81).671.25 (0.18-2.32).65

P02.07 (1.88-2.26)<.0012.42 (2.12-2.72)<.0010.97 (0.79-1.15).770.42 (0.23-0.61)<.001

P11.59 (1.43-1.76)<.0011.67 (1.43-1.91)<.0010.94 (0.75-1.14).570.60 (0.36-0.85).002

P21.36 (1.20-1.51)<.0011.39 (1.18-1.61)<.0011.06 (0.83-1.29).600.49 (0.26-0.72)<.001

P31.20 (1.05-1.34).0041.17 (0.97-1.37).080.90 (0.68-1.13).400.60 (0.31-0.89).01

P41.36 (1.20-1.52)<.0011.43 (1.19-1.66)<.0010.96 (0.73-1.18).700.69 (0.39-1.00).05

P51.21 (1.06-1.37).0041.08 (0.87-1.29).431.08 (0.82-1.35).540.84 (0.47-1.21).39

P61.26 (1.08-1.43).0011.24 (1.00-1.48).030.86 (0.62-1.10).250.35 (0.11-0.59)<.001

P71.19 (1.01-1.37).031.07 (0.83-1.31).551.12 (0.81-1.44).440.86 (0.43-1.29).53

P81.28 (1.08-1.48).0021.15 (0.88-1.42).240.97 (0.67-1.27).820.45 (0.13-0.77).001

P91.12 (0.91-1.33).250.97 (0.70-1.25).850.73 (0.43-1.03).080.37 (0.02-0.71)<.001

P101.05 (0.79-1.30).720.65 (0.36-0.93).050.75 (0.35-1.15).220.47 (0.03-0.97).04

P01.91 (1.75-2.06)<.0012.14 (1.90-2.38)<.0010.83 (0.69-0.97).020.85 (0.60-1.10).24

P11.40 (1.27-1.53)<.0011.50 (1.31-1.69)<.0011.08 (0.89-1.27).420.69 (0.44-0.94).02

P21.18 (1.06-1.30).0011.37 (1.20-1.55)<.0011.14 (0.92-1.35).210.60 (0.36-0.84).001

P31.24 (1.12-1.36)<.0011.16 (1.00-1.32).041.04 (0.84-1.24).690.82 (0.54-1.11).22

P41.22 (1.09-1.34)<.0011.21 (1.04-1.39).010.99 (0.79-1.19).940.80 (0.51-1.09).18

P51.12 (1.00-1.24).041.00 (0.84-1.16)>.991.20 (0.96-1.44).110.65 (0.39-0.92).01

P61.17 (1.04-1.30).011.08 (0.91-1.25).320.99 (0.78-1.20).940.92 (0.59-1.25).62

P71.04 (0.92-1.17).500.86 (0.70-1.02).110.88 (0.66-1.09).270.72 (0.41-1.03).08

P81.03 (0.90-1.16).650.99 (0.81-1.18).941.09 (0.82-1.36).530.51 (0.23-0.79).001

P91.03 (0.89-1.18).660.98 (0.78-1.17).831.12 (0.82-1.42).430.42 (0.15-0.69)<.001

P100.93 (0.76-1.09).390.84 (0.62-1.06).201.18 (0.79-1.57).370.39 (0.07-0.71)<.001

P02.10 (1.88-2.32)<.0011.93 (1.62-2.24)<.0010.76 (0.59-0.93).010.47 (0.28-0.67)<.001

P11.29 (1.12-1.46)<.0011.14 (0.91-1.36).201.15 (0.87-1.42).300.82 (0.49-1.14).27

P21.08 (0.92-1.23).311.00 (0.79-1.21).981.14 (0.84-1.43).380.70 (0.39-1.02).06

P31.00 (0.84-1.15).950.96 (0.74-1.18).731.22 (0.87-1.57).210.79 (0.41-1.17).29

P41.06 (0.88-1.23).510.99 (0.76-1.23).951.16 (0.81-1.51).370.78 (0.40-1.16).26

P50.87 (0.71-1.03).141.02 (0.77-1.27).891.30 (0.87-1.72).170.48 (0.15-0.80).002

P61.15 (0.94-1.36).130.98 (0.71-1.24).880.98 (0.64-1.31).880.56 (0.22-0.90).01

P70.97 (0.76-1.18).770.91 (0.63-1.20).561.29 (0.80-1.77).250.72 (0.24-1.20).25

P81.02 (0.78-1.26).880.86 (0.55-1.17).411.08 (0.61-1.54).750.88 (0.30-1.45).68

P90.90 (0.64-1.16).470.92 (0.56-1.28).681.13 (0.55-1.71).660.56 (0.01-1.11).12

P100.57 (0.30-0.83).020.54 (0.17-0.91).080.77 (0.01-1.55).571.55 (0.08-3.18).51

P02.10 (1.95-2.25)<.0012.35 (2.11-2.58)<.0010.83 (0.71-0.96).010.75 (0.56-0.95).01

P11.13 (1.03-1.24).011.29 (1.13-1.46)<.0011.05 (0.86-1.24).591.00 (0.69-1.30).98

P20.97 (0.87-1.07).550.95 (0.81-1.09).471.24 (1.01-1.46).040.99 (0.68-1.31).96

P30.89 (0.80-0.99).030.93 (0.79-1.06).310.91 (0.72-1.10).361.24 (0.87-1.62).21

P40.92 (0.82-1.01).100.89 (0.76-1.02).130.97 (0.77-1.17).750.92 (0.61-1.23).61

P50.80 (0.71-0.89)<.0010.80 (0.67-0.93).0091.02 (0.80-1.24).850.80 (0.51-1.10).19

P60.83 (0.74-0.92).0010.72 (0.60-0.84)<.0010.92 (0.72-1.12).450.65 (0.40-0.91).01

P70.83 (0.73-0.93).0010.75 (0.62-0.88).0010.89 (0.68-1.09).280.61 (0.35-0.87).003

P80.88 (0.77-0.98).040.76 (0.61-0.90).0030.92 (0.70-1.14).480.69 (0.40-0.98).04

P90.90 (0.78-1.02).110.73 (0.58-0.88).0030.86 (0.62-1.09).230.66 (0.35-0.98).04

P100.83 (0.69-0.97).030.68 (0.50-0.86).0040.94 (0.63-1.26).730.85 (0.41-1.29).52

aP0 is the time of portal adoption and P1-P10 stand for quarters 1-10 postadoption.

bLAB: laboratory.

cMESG: messaging.

dAPPT: appointment.

eSilent: being inactive.

Office Visits

For all the user subgroups, the office visit RRs were significantly larger than 1 within 6 months after portal adoption (RRs 1.13-2.1, 95% CIs 1.03-2.25, all P<.001) but were decreasing over time. The difference in office visit rates between the nonusers and most users—except for silent users—2 years postadoption was not significant. For the silent users, their office visit rates were not changed or were slightly lower by around 10% after 6 months postadoption. Patients who frequently used both messaging and laboratory functions had the largest RRs of office visits categorized as arrived, with an approximate 20% increase. Factoring in the APN of different user subgroups, for patients with fewer active health problems, their primary care service utilization was significantly lower after portal adoption. Meanwhile, with a heavy disease burden, the utilization was temporarily increased but was not significantly changed after 2 years postadoption.

Telephone Encounters

The change of telephone encounters was similar to that of office visits. Telephone encounters increased significantly at the time patients adopted portals (RRs 1.93-2.42, 95% CIs 1.62-2.72, all P<.001) and the RRs decreased over time. The silent users’ telephone encounters were significantly lower by 20% or more after 1 year postadoption (RRs 0.68-0.80, 95% CIs 0.50-0.93, all P<.01).

Appointment Cancellation

For all user subgroups, their cancellation rates were not significantly different to the nonusers and there was no trend in cancellation RRs over time.

Appointment No-Show

The no-show rates were significantly lower in most quarters postadoption: that of users were lower by 30% on average than nonusers and were not changed in the remaining quarters. In particular, patients using more messaging and messaging and laboratory combined had a larger reduction in no-show rates (average RR 0.61, minimum RR 0.35, P<.001). In summary, using patient portals is effective in reducing no-shows, but the relationship between portal usage and primary care service utilization is more complex than the simple substitution of online for in-person care.

Sensitivity Analysis

We analyzed the robustness of the results by changing the seeds used for conducting user subgroup clustering. This resulted in changes in user subgroup memberships: about 4.6% (95% CI 4.3-4.8) of patients had different memberships compared to the original clustering. The corresponding results were similar to the original model with respect to the overall RR trends for different outcome measures. The RR estimates and significance were slightly different but did not affect the major conclusions, which validates the robustness of this framework.

Principal Findings

Patient Portal Adoption

In terms of the characteristics of portal adopters, users were more likely to be female, white, married, and enrollees of the commercial insurance Blue Cross Blue Shield. Adoption disparities in gender, race, and socioeconomic status were observed, which is consistent with previous studies on social disparities in enrollment and use of patient portals (see Perzynski et al [27], Graetz et al [28], and Kruse et al [29] and the references therein). Surprisingly, instead of alienating the older generation, young adults aged 19-30 years tended not to adopt patient portals. Members of the younger generation are the habitual users of Web-based applications [30]. It suggests that being accustomed to using Internet and other Web portals may not be a powerful predictor of portal adoption. This is a seemingly counterintuitive observation and might be interpreted as young adults not being strongly motivated to use patient portals because they are healthier and have a relatively low level of health care consumption in general [31]. Being tech-savvy is not the driving force for portal adoption and subsequent usage.

Patient Portal Usage

The characteristics of portal adopters were not necessarily the same as those of active portal users. Our findings suggest that the people who potentially enjoy using or need to use patient portals are not aware of, or given enough access to, patient portals. The most important factor driving portal usage intensity is patient disease burden, measured by APN. Patients with a heavy disease burden would use clinical services more frequently and patient portals can provide more convenience for them. Unfortunately, the propensity of portal adoption among the high APN population was not high, although they might be the population that benefits the most from using patient portals. In addition, although Medicare patients did not show a strong intention to adopt portals [27-29,32,33], once they became users, they exhibited a relatively high level of utilization and they were not resistant to using the messaging function of portals for communication. Medicare patients may actually appreciate the value of patient portals but just have barriers to adopt it, which signals a lack of match in the patient portal market. External forces, such as incentives, reward programs, and policy initiatives, are needed to channel patients.

Understanding the unique needs and usage habits of different patient populations can contribute to a better and user-friendly design of the portal that can cater its service and functionality to patients’ various tastes and preferences. For instance, an important factor for predicting user types is age. Comparing the younger and the older generations, we found that their attitudes toward using portals to make an appointment or sending a message differed significantly. Older patients did not favor the appointment function as young people did, possibly because most of their appointments are follow-ups and are made directly after their office visits. However, older patients preferred to send messages to their providers compared to other age groups; the relatively high utilization demonstrated the value they found in messaging. This is possibly because they demand frequent and timely communication with their providers, and messaging is a good complement to telephone encounters and office visits to fulfill their heavy needs.

Lastly, the digital divide between races or ethnicities exists not only in adoption, but also in the subsequent use of portals. Nonwhite patients, in general, tended not to adopt nor actively use a portal. In particular, black and African American patients tended not to adopt a portal, and Hispanic patients were very inactive after adopting portals. In particular, Asian patients exhibited a low level of utilization of the messaging function, implying a language barrier [28,34]. It was also found in other research that racial and ethnic minority groups, especially, reported concerns about privacy and information security and they differed from the advantaged (ie, high socioeconomic status) groups in their knowledge and skills of, and comfort in using, the technology, in addition to their accessibility to the technology infrastructure [6,35-37].

Language barriers, poor health literacy, and a low socioeconomic status, among other barriers, contribute substantially to the digital divide. Addressing these barriers will require patient education, infrastructure enhancement, as well as the technological designs that enable patients to communicate with providers in a secure and convenient way [6]. Providers, especially those serving vulnerable populations, should communicate with patients about portal usage and take time to discuss and demonstrate the technology, such as how to use different portal functions. Policy makers and technology developers should ensure the security, privacy, and ease of use of patient portals and the telehealth infrastructure, factoring in the special needs and the concerns of racial and ethnic minority groups. The heterogeneous adoption and usage behaviors of patients signal that the technology acceptance by people is not uniform and can be compounded by multiple factors, such as the conditions to facilitate the use, the ease of use, and the perceived usefulness [38]. Technology adoption theory would play an important role in guiding the design and development of portal functions that benefit patients with different characteristics and care needs.

Care Service Utilization and Appointment Adherence

First, the portal usage effects are heterogeneous rather than homogeneous. Different user groups behaved in different ways; ignoring such heterogeneity could lead to misspecification of such effects. Patients who frequently used messaging and laboratory together had the largest increase in primary care service utilization, including office visits and telephone encounters, while silent portal users had the largest reduction in using primary care services compared to before adoption. Mixing the two groups will lead to a possible conclusion that the primary care service utilization has not changed, as the positive and negative effects can get cancelled out.

Second, the portal usage effects are dynamic rather than static. Therefore, it is necessary to conduct studies using longitudinal data, not solely relying on observations from cross-sectional studies. The trends of office visit and telephone encounter RRs suggest that the convenience brought by patient portals for supporting better provider-patient interaction might reduce patients’ in-person visits over a longer time frame rather than immediately. This may be due to the fact that patients need time to adapt to portal functionalities, and patient portals influence patients’ health behavior gradually. Both the short-term (ie, a temporary boost) and the long-term (ie, a gradual decline) impacts are critical to informing service operations and guiding policy decisions. Whereas the portal usage was not shown to significantly reduce clinical service consumption immediately, portal activities, such as replying to secure messages, would inevitably increase the provider’s service time. Thus, exploring the payment structure that accommodates the technologically mediated interactions between providers and patients (eg, text messaging, emails, and virtual visits) is instrumental to gaining the buy-in of providers. Policy makers and payers must accordingly recognize and value the amount of time providers spend on interacting and educating patients, particularly the vulnerable and disadvantaged ones, both online and offline.

Lastly, the correct understanding of the heterogeneous and dynamic property of portal usage effects will enable us to carry out targeted and proactive interventions to achieve better patient outcomes. While the user subgroups behaved differently toward their health care consumption, the no-show rates of portal users were, overall, lower compared to that of nonusers, with different magnitudes of change. It also reveals that actively using patient portals, in contrast to being silent, leads to a larger improvement in appointment adherence. To achieve better patient engagement, providers can take the initiative in messaging patients, especially the ones with a high APN, to stimulate their portal usage and, thus, to raise their awareness of care engagement. In addition, elements of gamification can be imbedded into portal functions to encourage and reward patients. Virtual rewards or incentives can be made to patients who exhibit a high level of portal interactions (eg, actively reading the after-visit summary, physicians’ notes, and lab results, as well as participating in portal-based surveys, such as quality-of-care questionnaires and patient-reported outcomes).


There are several limitations to our study. First, our causal inference analysis was based on an observational study. Admittedly, no unmeasured confounder is typically assumed to identify causal effects and is difficult to validate. However, even with unmeasured confounders, as long as they are time-invariant, their effects will be “cancelled” owing to the DID study design with the before-after comparison. If confounders are time-varying and measurable, we can treat them in the same fashion as dealing with a patient’s APN (ie, a time-varying confounder) and their disease process (ie, conditioning on their APN and whether a new disease occurred at each quarter). To test the robustness of our framework against unmeasured time-varying confounders, in future work, we plan to develop a trend-in-trends study [39]. In addition, we propose to design a synthetic control test [40] as a sensitivity analysis to evaluate the impact of any unmeasured confounders to our major results. Second, the dataset was limited to UF Health patients. We presented the demographics and socioeconomic characteristics of our patient population as a one-site study so that the results can be compared to other systems with the same or a different patient makeup. We also examined the institutional evaluation and management codes, which represent the severity level of patients: the larger the number, the more complicated the service [41]. It confirmed that our patient population shared a similar patient level in terms of severity to other academic health systems. Therefore, we hope that this patient population can be considered as being representative to generate a general insight. Third, the causal effects of portal usage on patient-reported outcomes and other adherence behaviors (eg, medication adherence) cannot be established using the data collected for this study. Also, we were not able to assess the impact of portal usage on patients’ specialty care consumption and their out-of-the-network urgent care and emergency department visits. We plan to expand the data spectrum to include some of these outcomes as part of the future work. Lastly, the business value and economic impact of patient portal implementation need to be quantified.


In closing, patients differ in their portal adoption and usage behaviors, and the portal effects are heterogeneous and dynamic. There exists a lack of match in the patient portal market in the sense that patients who benefit the most from using patient portals are not actively adopting patient portals. Patient portal usage was confirmed as effective in reducing appointment no-shows. However, to maximize the potential of patient portals, it is paramount to understanding the value that patient portals could bring to patients who have exhibited different characteristics and care needs. Health care delivery planners and administrators should, on the one hand, remove the barriers of adoption for the portal beneficiaries and, on the other hand, incorporate the impact of portal usage into care coordination and workflow design, ultimately aligning patients’ and providers’ needs and functionalities to enhance care delivery.


Publication of this article was funded in part by the University of Florida Open Access Publishing Fund.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Illustration of two-stage clustering and descriptive statistics of the clustering results.

DOCX File , 131 KB


  1. Powell K, Myers C. Online Journal of Nursing Informatics (OJNI). 2018 Feb 27. Electronic patient portals: Patient and provider perceptions   URL: [accessed 2019-04-03] [WebCite Cache]
  2. Marcotte L, Seidman J, Trudel K, Berwick DM, Blumenthal D, Mostashari F, et al. Achieving meaningful use of health information technology: A guide for physicians to the EHR incentive programs. Arch Intern Med 2012 May 14;172(9):731-736. [CrossRef] [Medline]
  3. Woods SS, Schwartz E, Tuepker A, Press NA, Nazi KM, Turvey CL, et al. Patient experiences with full electronic access to health records and clinical notes through the My HealtheVet Personal Health Record Pilot: Qualitative study. J Med Internet Res 2013 Mar 27;15(3):e65 [FREE Full text] [CrossRef] [Medline]
  4. Gordon NP, Hornbrook MC. Differences in access to and preferences for using patient portals and other eHealth technologies based on race, ethnicity, and age: A database and survey study of seniors in a large health plan. J Med Internet Res 2016 Mar 04;18(3):e50 [FREE Full text] [CrossRef] [Medline]
  5. Sorondo B, Allen A, Fathima S, Bayleran J, Sabbagh I. Patient portal as a tool for enhancing patient experience and improving quality of care in primary care practices. EGEMS (Wash DC) 2016;4(1):1262 [FREE Full text] [CrossRef] [Medline]
  6. Anthony DL, Campos-Castillo C, Lim PS. Who isn't using patient portals and why? Evidence and implications from a national sample of US adults. Health Aff (Millwood) 2018 Dec;37(12):1948-1954. [CrossRef] [Medline]
  7. Tai-Seale M, Downing N, Jones V, Milani R, Zhao B, Clay B, et al. Technology-enabled consumer engagement: Promising practices at four health care delivery organizations. Health Aff (Millwood) 2019 Mar;38(3):383-390. [CrossRef] [Medline]
  8. Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. JAMA 2012 Nov 21;308(19):2012-2019. [CrossRef] [Medline]
  9. Otte-Trojel T, de Bont A, Rundall TG, van de Klundert J. How outcomes are achieved through patient portals: A realist review. J Am Med Inform Assoc 2014;21(4):751-757 [FREE Full text] [CrossRef] [Medline]
  10. Zhou Y, Leith W, Li H, Tom JO. Personal health record use for children and health care utilization: Propensity score-matched cohort analysis. J Am Med Inform Assoc 2015 Jul;22(4):748-754. [CrossRef] [Medline]
  11. Leveille SG, Mejilla R, Ngo L, Fossa A, Elmore JG, Darer J, et al. Do patients who access clinical information on patient Internet portals have more primary care visits? Med Care 2016 Jan;54(1):17-23. [CrossRef] [Medline]
  12. Bavafa H, Hitt LM, Terwiesch C. The impact of e-visits on visit frequencies and patient health: Evidence from primary care. Manage Sci 2018 Dec;64(12):5461-5480. [CrossRef]
  13. North F, Crane SJ, Chaudhry R, Ebbert JO, Ytterberg K, Tulledge-Scheitel SM, et al. Impact of patient portal secure messages and electronic visits on adult primary care office visits. Telemed J E Health 2014 Mar;20(3):192-198 [FREE Full text] [CrossRef] [Medline]
  14. Meng D, Palen T, Tsai J, McLeod M, Garrido T, Qian H. Association between secure patient-clinician email and clinical services utilisation in a US integrated health system: A retrospective cohort study. BMJ Open 2015 Nov 09;5(11):e009557 [FREE Full text] [CrossRef] [Medline]
  15. Ammenwerth E, Schnell-Inderst P, Hoerbst A. The impact of electronic patient portals on patient care: A systematic review of controlled trials. J Med Internet Res 2012 Nov 26;14(6):e162 [FREE Full text] [CrossRef] [Medline]
  16. Mold F, de Lusignan S, Sheikh A, Majeed A, Wyatt JC, Quinn T, et al. Patients' online access to their electronic health records and linked online services: A systematic review in primary care. Br J Gen Pract 2015 Mar;65(632):e141-e151 [FREE Full text] [CrossRef] [Medline]
  17. Irizarry T, DeVito Dabbs A, Curran CR. Patient portals and patient engagement: A state of the science review. J Med Internet Res 2015 Jun 23;17(6):e148 [FREE Full text] [CrossRef] [Medline]
  18. Horvath M, Levy J, L'Engle P, Carlson B, Ahmad A, Ferranti J. Impact of health portal enrollment with email reminders on adherence to clinic appointments: A pilot study. J Med Internet Res 2011 May 26;13(2):e41 [FREE Full text] [CrossRef] [Medline]
  19. Lin C, Mistry N, Boneh J, Li H, Lazebnik R. Text message reminders increase appointment adherence in a pediatric clinic: A randomized controlled trial. Int J Pediatr 2016;2016:8487378 [FREE Full text] [CrossRef] [Medline]
  20. Mendel A, Chow S. Impact of health portal enrolment with email reminders at an academic rheumatology clinic. BMJ Qual Improv Rep 2017;6(1):1-9 [FREE Full text] [CrossRef] [Medline]
  21. Bergmo TS, Kummervold PE, Gammon D, Dahl LB. Electronic patient-provider communication: Will it offset office visits and telephone consultations in primary care? Int J Med Inform 2005 Sep;74(9):705-710. [CrossRef] [Medline]
  22. Zhou YY, Garrido T, Chin HL, Wiesenthal AM, Liang LL. Patient access to an electronic health record with secure messaging: Impact on primary care utilization. Am J Manag Care 2007 Jul;13(7):418-424 [FREE Full text] [Medline]
  23. Hornik K, Feinerer I, Kober M, Buchta C. Spherical k-means clustering. J Stat Softw 2012 Sep;50(10):1-22 [FREE Full text] [CrossRef]
  24. Lopez M, Gutman R. Estimation of causal effects with multiple treatments: A review and new ideas. Stat Sci 2017 Aug;32(3):432-454. [CrossRef]
  25. Zhong X, Liang M, Sanchez R, Yu M, Budd PR, Sprague JL, et al. On the effect of electronic patient portal on primary care utilization and appointment adherence. BMC Med Inform Decis Mak 2018 Oct 16;18(1):84 [FREE Full text] [CrossRef] [Medline]
  26. Medicare Interactive. Medicare eligibility for those under 65   URL: https:/​/www.​​get-answers/​medicare-basics/​medicare-eligibility-overview/​medicare-eligibility-for-those-under-65 [accessed 2019-04-03]
  27. Perzynski AT, Roach MJ, Shick S, Callahan B, Gunzler D, Cebul R, et al. Patient portals and broadband Internet inequality. J Am Med Inform Assoc 2017 Sep 01;24(5):927-932 [FREE Full text] [CrossRef] [Medline]
  28. Graetz I, Gordon N, Fung V, Hamity C, Reed ME. The digital divide and patient portals: Internet access explained differences in patient portal use for secure messaging by age, race, and income. Med Care 2016 Aug;54(8):772-779. [CrossRef] [Medline]
  29. Kruse CS, Argueta DA, Lopez L, Nair A. Patient and provider attitudes toward the use of patient portals for the management of chronic disease: A systematic review. J Med Internet Res 2015 Feb 20;17(2):e40 [FREE Full text] [CrossRef] [Medline]
  30. Friemel TN. The digital divide has grown old: Determinants of a digital divide among seniors. New Media Soc 2014 Jun 12;18(2):313-331. [CrossRef]
  31. Mason CN, Miller T. International projections of age specific healthcare consumption: 2015-2060. J Econ Ageing 2018 Nov;12:202-217. [CrossRef]
  32. Levy H, Janke AT, Langa KM. Health literacy and the digital divide among older Americans. J Gen Intern Med 2015 Mar;30(3):284-289 [FREE Full text] [CrossRef] [Medline]
  33. Hong YA, Cho J. Has the digital health divide widened? Trends of health-related Internet use among older adults from 2003 to 2011. J Gerontol B Psychol Sci Soc Sci 2017 Sep 01;72(5):856-863. [CrossRef] [Medline]
  34. Tieu L, Sarkar U, Schillinger D, Ralston JD, Ratanawongsa N, Pasick R, et al. Barriers and facilitators to online portal use among patients and caregivers in a safety net health care system: A qualitative study. J Med Internet Res 2015 Dec 03;17(12):e275 [FREE Full text] [CrossRef] [Medline]
  35. Campos-Castillo C, Anthony DL. The double-edged sword of electronic health records: Implications for patient disclosure. J Am Med Inform Assoc 2015 Apr;22(e1):e130-e140. [CrossRef] [Medline]
  36. Stablein T, Hall J, Pervis C, Anthony D. Negotiating stigma in health care: Disclosure and the role of electronic health records. Health Sociol Rev 2015 Sep 07;24(3):227-241. [CrossRef]
  37. Miller DP, Latulipe C, Melius KA, Quandt SA, Arcury TA. Primary care providers' views of patient portals: Interview study of perceived benefits and consequences. J Med Internet Res 2016 Jan 15;18(1):e8 [FREE Full text] [CrossRef] [Medline]
  38. Hoogenbosch B, Postma J, de Man-van Ginkel JM, Tiemessen NA, van Delden JJ, van Os-Medendorp H. Use and the users of a patient portal: Cross-sectional study. J Med Internet Res 2018 Sep 17;20(9):e262 [FREE Full text] [CrossRef] [Medline]
  39. Ji X, Small DS, Leonard CE, Hennessy S. The trend-in-trend research design for causal inference. Epidemiology 2017;28(4):529-536. [CrossRef]
  40. Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. J Am Stat Assoc 2010 Jun;105(490):493-505. [CrossRef]
  41. American Academy of Family Physicians (AAFP). Coding for evaluation and management services   URL: [accessed 2019-12-17]

APN: active problem number
APPT: appointment
DID: difference-in-differences
EMR: electronic medical record
HINTS: Health Information National Trends Survey
LAB: laboratory
MED: medication
MESG: messaging
MLR: multinomial logit regression
N/A: not applicable
OR: odds ratio
Q1: first quarter, January-March
Q2: second quarter, April-June
Q3: third quarter, July-September
Q4: fourth quarter, October-December
RR: rate ratio
UF: University of Florida

Edited by G Eysenbach; submitted 16.04.19; peer-reviewed by J Hefner, T Irizarry, B Smith; comments to author 03.10.19; revised version received 07.11.19; accepted 12.11.19; published 25.02.20


©Xiang Zhong, Jaeyoung Park, Muxuan Liang, Fangyun Shi, Pamela R Budd, Julie L Sprague, Marvin A Dewar. Originally published in the Journal of Medical Internet Research (, 25.02.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.