Epic Systems is a major provider of health information technology in the United States, providing electronic medical records for more than 250 million patients [1]. Epic’s platform includes predictive models for patient care, including a model that predicts a patient’s probability of being a no-show for an outpatient health care appointment. Epic reports the area under the receiver operating characteristic curve (AUC) of this model to range between 0.71 to 0.81, but the model has not been externally validated in certain groups of patients, including people with HIV [2]. Previous external validation of Epic’s proprietary sepsis prediction model showed reduced performance [3].

Regular medical care for people with HIV is of the utmost importance [4,5]. Missed medical appointments among people with HIV are independently associated with increased mortality [6]. Additionally, HIV is a disease of extreme disparity, with persons of racial and sexual minorities disproportionately impacted [7]. A no-show predictive model among people with HIV could improve clinical care by allowing providers to proactively engage patients at high risk of missing appointments. We examined the performance of Epic’s no-show model among people with HIV at an academic medical center and whether the performance was impacted by the addition of demographic and HIV clinical information.

We identified 674 people with HIV who contributed 1406 total scheduled in-person appointments during the study period. The performance of the Epic model among people with HIV for all appointments in any outpatient clinic had an AUC of 0.65 (95% CI 0.63-0.66). When we restricted the data to include only HIV care clinic appointments, we identified 331 people with HIV who contributed 440 infectious disease appointments. The AUC of the Epic model for HIV care appointments among people with HIV was 0.63 (95% CI 0.59-0.67), and there was no significant difference in performance compared to the model that included all appointments (P=.36). The alternate model we created for people with HIV attending HIV care appointments had an AUC of 0.78 (95% CI 0.75-0.82), a significant improvement over the Epic model restricted to HIV care appointments (P<.001; Figure 1). Features identified as important in this model included ones from the original Epic model, such as appointment lead time and appointment length. However, some of the new demographic and clinical features were also identified as important, such as viral load >200 copies per mL, lower CD4 T cell counts (both 50 to <200 cells/mm³ and 200 to <350 cells/mm³), and female sex.

We examined the performance of the Epic no-show predictive model among people with HIV using all appointments regardless of specialty and among HIV care appointments at the infectious disease clinic. For both models among people with HIV, performance was significantly lower than reported by Epic.

We found that a model that incorporated a subset of the features used in the original Epic model along with demographic and HIV clinical information performed better among people with HIV attending HIV care appointments. Similar to previous studies, our model showed that HIV clinical information (CD4 count and HIV viral load laboratory test results) are important for predicting future appointment attendance in people with HIV [10,11]. The inclusion of demographic factors further improved model performance.

A limitation of this project was our inability to fully replicate all 22 features from Epic’s no-show prediction model due to the proprietary nature of the model. We were able to identify seven features used by Epic and include these as well as additional clinical and demographic features in the alternate model to see if the additional features would improve the model performance among people with HIV. Two of the model features (race and sex) added in our alternate model were originally part of version 1 of the Epic no-show model but were later removed due to concerns about adverse impacts on marginalized groups [2].

The improvement in the performance of the alternate model over the proprietary Epic model demonstrates that among populations with extreme disparities, such as people with HIV, the inclusion of demographic information may enhance the prediction of appointment attendance. The alternate model further reveals that the prediction of appointment attendance in people with HIV can be improved by using HIV clinical information such as CD4 count and HIV viral load test results as features in the model. Hospitals and electronic medical record systems interested in developing or improving their no-show prediction models among diseases with high patient disparity may benefit from including these features.