We performed a retrospective analysis of electronic health record (EHR) data from 45 practices within a large pediatric primary care network. These practices are certified as patient-centered medical homes by The Joint Commission; together, they provide care for approximately 315,000 children throughout Western Pennsylvania across 13 counties. All the practices shared one EHR, which offered embedded video visits through a patient portal. Some of these practices had briefly trialed a model of acute care telemedicine in 2015; however, none of the practices were offering telemedicine services at the start of the pandemic.

Local payers began to offer payment for telemedicine when the patient is at home on March 17, 2020 [17]. On March 23, 2020, the first county-specific stay-at-home order was issued in Pennsylvania [17,18]. Several practice leads trialed multiple telemedicine platforms and workflows from March 18-20, 2020, with implementation strategies shared with all practices via videoconference on March 23, 2020. The network quality and safety leaders led collaborative learning videoconferences two to three times per week for the next two months with all physicians, advanced practice providers (APPs), and practice managers, sharing telemedicine best practices and discussing other COVID-19–related topics. Initial relaxation of the stay-at-home order for the largest metropolitan county occurred on May 15, 2020, with transition of the state to its “yellow phase.”

We obtained encounter data for all telemedicine and in-person visits between March 18 and May 2, 2020, from the EHR, and we identified patient panels for each practice, defined as all patients with one or more encounters at the practice in the prior two years. For each practice’s patient panel, patient demographics, date of last preventive visit, and counts of telemedicine, office, ED, and urgent care visits were obtained from the EHR. To complement this EHR-based evaluation, we also surveyed the primary care clinicians across the practice network regarding the usability, usefulness, and perception of patient and clinician experience of telemedicine.

For each practice in the network, the number of practice clinicians (doctors and APPs) and practice site locations were obtained from network records. Practice site counties were classified as rural or urban using the 2013 rural-urban continuum codes [19].

Telemedicine visits were identified using EHR encounter type codes. All completed telemedicine visits from March 18 to May 2, 2020, were included. For each telemedicine visit, we extracted the practice site, age of the child on the date of visit, and primary visit diagnosis.

Telemedicine visit primary diagnoses were categorized based on International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnosis codes into 22 broad ICD-10-CM diagnostic categories corresponding to organ systems [20]. For nonspecific categories (eg, “symptoms, signs, and abnormal clinical and laboratory findings, not elsewhere classified” (R00-R99)), we reviewed the subcategories and recategorized them into the relevant organ system. For example, the subcategory of “symptoms and signs involving the skin and subcutaneous tissue” (R20-R23) was grouped with “diseases of the skin and subcutaneous tissues” (L00-L99) into “skin and subcutaneous tissue diagnoses.”

To compare the volumes of telemedicine visits with other modes of care delivery, we extracted each practice’s volumes from the EHR, including in-person office, urgent care, and ED visits, as well as telephone encounters (excluding those with a telemedicine or in-person visit on the same day) during a 2-week window during the pandemic (April 19 to May 2, 2020).

Finally, for each child identified as part of a practice’s panel, we extracted age, race and ethnicity, health insurance (Medicaid vs commercial), and whether the child had a preventive visit within the prior 12 months. The child’s race and ethnicity were originally recorded in the EHR based on parent response during the child’s first visit with the practice. Across all practices, 82% of patients were identified as White non-Hispanic; therefore, analyses by patient race and ethnicity were limited to comparing children identified as White non-Hispanic to children identified as any other racial or ethnic identity, of which the majority identified as “other” (9%), non-Hispanic Black (8%), or Hispanic (1%).

To complement this primarily EHR-based analysis, clinicians in the primary care network were invited to participate in a web-based survey. The survey items examined the usability and usefulness of telemedicine through items modified from the Technology Usability Questionnaire, which is a validated survey incorporating questions from the Technology Acceptance Model, Telemedicine Satisfaction Questionnaire, and Post-Study System Usability Questionnaire and encompasses five subscales: usefulness, ease of use, effectiveness, reliability, and satisfaction, with all items using Likert scales [21]. We added questions pertaining to the physician experience of telemedicine, including perceived impact on quality of care (informed by the Institute of Medicine’s six domains of quality), impact on job satisfaction (informed by self-determination theory), and perceived usefulness of telemedicine for different visit reasons [22,23]. The survey included 37 questions, and it is available in Multimedia Appendix 1. Participants had the option to identify their practice or leave this item blank.

Clinicians were invited to participate in the anonymous web-based survey from April 28 to May 14, 2020, through a series of 4 emails. The timeframe was chosen to capture summative attitudes and experiences of clinicians coinciding with the end of the EHR-based analysis.

For each practice, to account for variation in practice size, we determined the total number of telemedicine visits completed per week and divided the number of visits by the number of active patients in that practice to provide a standardized rate of telemedicine visits per 1000 patients. To categorize high versus low telemedicine use practices, the rate of telemedicine visits per 1000 patients per week was averaged for the 2-week period (April 19 to May 2, 2020) occurring after the first month of telemedicine implementation. This time frame was chosen to categorize practices at a time where telemedicine visit volume had stabilized so that the analysis could focus on practices with high versus low use (as opposed to early vs late adopters). The 45 practices were categorized into tertiles based on their telemedicine visit rates, which we labeled as low, intermediate, or high telemedicine use practices.

The analyses compared practice-level characteristics, telemedicine visit diagnoses, in-person visit volumes, variation in volume by patient characteristics, and clinician attitudes across telemedicine use tertiles. Across practice-level telemedicine use tertiles, we compared practice-level characteristics and in-person visits per 1000 patients per week using Kruskal-Wallis tests. We compared the percentage of telemedicine visits within each individual diagnosis category across tertiles using logistic regression.

For patients in each practice’s panel, we determined the percentage of children who had one or more telemedicine visits during the 2-week period of the focused analysis (April 19 to May 2, 2020) by specific child characteristics (age, race and ethnicity, insurance type, and receipt of a preventive visit in the prior year). First, to assess whether there was a significant difference in telemedicine use by child characteristics across all practices (regardless of practice-level telemedicine use), we used a child-level logistic regression model clustered by practice but with no tertile variable. To determine whether increased practice-level use of telemedicine altered the differences in telemedicine use by child characteristics, we assessed the significance of an interaction term between each child characteristic category and practice tertile within a series of child-level logistic regression models clustered by practice across all tertiles. We tested the significance of the interaction terms using Wald tests, and we present the results as adjusted percentages determined through these models.

Clinician survey responses were compared across tertiles using linear regression clustered by practice, excluding respondents who did not identify their practice. We also compared responses for respondents who did not identify their practice versus those who did, again using linear regression, and found no significant differences.

All analyses were conducted in Stata version 16.1 (StataCorp) with significance assessed using an alpha level of .05.

This analysis was part of a quality improvement project aimed at improving pediatric primary care telemedicine delivery and was approved by the University of Pittsburgh Medical Center’s Quality Review Committee. Projects approved by this committee do not meet the definition of human subjects research and therefore do not require formal approval by an institutional review board.

High telemedicine use practices had more physicians in the practice (median 4) than low telemedicine use practices (median 3, P=.04); however, all other practice-level characteristics were similar (Table 1).

Across all tertiles, telemedicine visits were most common for mental health and skin/soft tissue–related diagnoses (Table 2). However, the percentage of visits in each of these diagnostic categories varied with practice-level telemedicine use. Visits with skin-related diagnoses, for example, comprised 17.5% of telemedicine visits (467/2661) at low telemedicine use practices, and 15.0% of visits (1435/9587) at high telemedicine use practices (P=.006). Although visits for skin-related diagnoses represented a smaller percentage of telemedicine visits at high telemedicine use practices, the number of skin-related visits per 1000 patients per week was higher at high telemedicine use practices compared to low telemedicine use practices (median 14 vs 7 telemedicine skin related visits per 1000 patients per week at high and low telemedicine use practices, respectively; P<.001). High telemedicine use practices had larger percentages of telemedicine visits devoted to respiratory (P<.001), ear/mastoid (P<.001), and genitourinary (P=.02) diagnoses than low telemedicine use practices (Table 2).

Compared to low telemedicine use practices, practices with high telemedicine use had fewer in-person office visits (median 10 vs 16 in-person office visits per 1000 patients per week at high vs low telemedicine use practices, respectively; P=.005; Table 3). Practices with high telemedicine use also had slightly more ED visits (median 2 vs 1 ED visits per 1000 patients per week at high and low telemedicine use practices, respectively; P=.02) but similar urgent care visits. When accounting for both in-person and telemedicine office visits, high telemedicine use practices had more total primary care encounters per 1000 patients per week (median 28 vs 22 encounters, P=.006). Of note, among high telemedicine use practices, total primary care encounters (in-person and telemedicine) represented a median of 53% of the weekly volume in 2019. In contrast, among low telemedicine use practices, total primary care encounters represented a median of 46% of the 2019 weekly volume. Telephone encounters occurring separate from a visit were similar across tertiles.

In logistic regression models without practice-level tertile designation, use of telemedicine varied significantly by child race and ethnicity (P<.001), child age (P<.001), and receipt of preventive care in the prior 12 months (P<.001) but did not vary by child insurance category (P=.40). However, the interaction terms between the practice tertile and each of these characteristics were not significant (Table 4). This indicates that although differences exist in the full sample for telemedicine use by child race and ethnicity, child age, and recency of a preventive visit, increasing practice-level use of telemedicine did not change the relationship between these patient characteristics and the likelihood of a telemedicine visit.

The survey was completed by 121 clinicians, including 88 who identified their practice and 33 who did not (34% response rate). Clinician attitudes regarding the usability, usefulness for child health, usefulness for clinician experience, or impact of telemedicine did not vary significantly across tertiles (Table 5).

Among a network of pediatric practices, we examined practice-level variation in use of telemedicine during the COVID-19 pandemic, allowing us to explore ongoing questions about the relationship between telemedicine use, receipt of care, and equity in receipt of care. We found that increased practice-level telemedicine use was associated with more physicians in the practice, more varied telemedicine encounter diagnoses, and fewer in-person office visits per 1000 patients per week.

While concerns exist that telemedicine may promote overutilization, we found that in the context of the pandemic, greater pediatric primary care practice-level telemedicine use did not result in overutilization but rather supported primary care encounter volume slightly closer to usual rates during a time when underutilization of primary care and other health care settings was prevalent [24-28]. We also found that high telemedicine use practices had slightly more ED visits than low telemedicine use practices during the study period. Again, this finding must be interpreted in the setting of overall decreased health care encounters during this time, such that this slight increase could represent either a greater ability of telemedicine to triage children to emergent care during the COVID-19 pandemic or, alternatively, a slightly greater need for emergent care for patients in these practices. Another possibility is that high telemedicine use practices may have adhered more strictly to guidelines to reduce in-person office visits; that goal, rather than telemedicine itself, may have led to the slight increase in ED visits.

In addition to concerns about overutilization, concerns have been raised about whether increased telemedicine use will improve equity of access to care or exacerbate disparities in access due to the digital divide [2,29,30]. In our sample, children without a preventive visit in the past year and children who identified as races or ethnicities other than White non-Hispanic were less likely to have had a telemedicine visit, while telemedicine use did not vary by child insurance type. These differences in the overall population use of telemedicine build upon prior concerns about differences in access to broadband internet, patient portals, and telemedicine [29,31-34]. However, the nonsignificant interaction terms indicate that increased practice-level use of telemedicine services during this specific period neither reduced nor increased these differences. In cases where telemedicine is intended to be a tool to improve equity in access to care, these results indicate that simply increasing the use of telemedicine may not be sufficient to ensure more equitable access. Indeed, telemedicine has the potential to exchange one set of barriers to care (transportation issues, time constraints, hidden costs of missing work) for another (need for internet access, device capability, computer literacy). For telemedicine to more effectively reduce disparities in access, it will be necessary to implement telemedicine in ways that more intentionally overcome barriers.

While increased use of telemedicine by practices did not translate into greater equity in telemedicine use for patients, it did translate into more varied telemedicine use based on visit diagnoses. Additionally, we observed an increase in the proportion of visits related to respiratory and ear, nose, and throat (ENT) symptoms. Concerns have been raised about assessments of ears in the absence of tele-otoscopy, which was not available for the studied visits [35]. Given that clinicians in low versus high telemedicine use tertiles reported similar views about the suitability of telemedicine for acute care, increased respiratory and ENT diagnoses at high telemedicine use practices may reflect a stronger practice-wide commitment to reducing in-person visits due to the COVID-19 pandemic as opposed to increased comfort with caring for these diagnoses via telemedicine. For all practices, regardless of their use tertile, the most common telemedicine visit diagnosis categories were mental health and skin-related diagnoses. This differs from the most common diagnoses for visits by children to commercial direct-to-consumer telemedicine, where the most common visit diagnosis category was nose/sinus infections [4]. In contrast, the most common diagnosis group among these primary care telemedicine visits was mental health, which accounted for approximately 30% of telemedicine visits across the full set of practices.

One key limitation of this EHR analysis is we cannot account for patient or parent preferences for or satisfaction with telemedicine use. Additionally, the racial and ethnic diversity within our sample was minimal, limiting analyses by race and ethnicity to a comparison of White non-Hispanic children and children identified as any other race or ethnicity. We also sought to compare visit volume by parental language; however, limited numbers of children with parental preference for non-English languages were identified in the relevant EHR field (0.5%). Although systems are in place to integrate ED and urgent care visit information into the EHR, ED and urgent care visits outside of our health system may still have been missed. However, we do not anticipate that this would result in any systematic bias across the practice tertiles given the large number of integrated ED and urgent care centers across the region. We also note that we focused this analysis on telemedicine visit diagnoses and volumes and did not examine quality of care or clinical outcomes. Finally, this analysis focused on telemedicine use within a specific context; therefore, the generalizability of the findings will need to be examined through other sources.

This study demonstrates that a large pediatric primary care network rapidly integrated use of telemedicine when given a favorable payment environment and public health mandates. The integration of telemedicine allowed continued contact with patients during the pandemic, largely for mental health care, with high practice-level telemedicine use allowing for more encounters with patients per week during a time where underutilization of primary care was common. Further work is needed to understand the sustainability of the pandemic-related surge in primary care telemedicine use and to identify ways to enhance the ability of telemedicine to improve access for those with access barriers.