This paper is in the following e-collection/theme issue:

Clinical Informatics (918) Machine Learning (1400) HIV/AIDS/STI Prevention and Care (823) Clinical Information and Decision Making (1339) Electronic Health Records (971)

Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/43277, first published .
No-show Prediction Model Performance Among People With HIV: External Validation Study

No-show Prediction Model Performance Among People With HIV: External Validation Study

No-show Prediction Model Performance Among People With HIV: External Validation Study

Authors of this article:

Joseph A Mason1 Author Orcid Image ;   Eleanor E Friedman1 Author Orcid Image ;   Juan C Rojas2 Author Orcid Image ;   Jessica P Ridgway1 Author Orcid Image
Article Authors Cited by (1) Tweetations (5) Metrics

    Short Paper

    1The Chicago Center for HIV Elimination, Department of Medicine, University of Chicago, Chicago, IL, United States

    2Division of Pulmonary & Critical Care Medicine, Department of Medicine, Rush University Medical Center, Chicago, IL, United States

    Corresponding Author:

    Joseph A Mason, BS

    The Chicago Center for HIV Elimination

    Department of Medicine

    University of Chicago

    5841 S Maryland Ave

    Chicago, IL, 60637

    United States

    Phone: 1 773 702 9586

    Email: joemason@bsd.uchicago.edu


    Background: Regular medical care is important for people living with HIV. A no-show predictive model among people with HIV could improve clinical care by allowing providers to proactively engage patients at high risk of missing appointments. Epic, a major provider of electronic medical record systems, created a model that predicts a patient’s probability of being a no-show for an outpatient health care appointment; however, this model has not been externally validated in people with HIV.

    Objective: We examined the performance of Epic’s no-show model among people with HIV at an academic medical center and assessed whether the performance was impacted by the addition of demographic and HIV clinical information.

    Methods: We obtained encounter data from all in-person appointments among people with HIV from January 21 to March 30, 2022, at the University of Chicago Medicine. We compared the predicted no-show probability at the time of the encounter to the actual outcome of these appointments. We also examined the performance of the Epic model among people with HIV for only HIV care appointments in the infectious diseases department. We further compared the no-show model among people with HIV for HIV care appointments to an alternate random forest model we created using a subset of seven readily accessible features used in the Epic model and four additional features related to HIV clinical care or demographics.

    Results: We identified 674 people with HIV who contributed 1406 total scheduled in-person appointments during the study period. Of those, we identified 331 people with HIV who contributed 440 HIV care appointments. The performance of the Epic model among people with HIV for all appointments in any outpatient clinic had an area under the receiver operating characteristic curve (AUC) of 0.65 (95% CI 0.63-0.66) and for only HIV care appointments had an AUC of 0.63 (95% CI 0.59-0.67). The alternate model we created for people with HIV attending HIV care appointments had an AUC of 0.78 (95% CI 0.75-0.82), a significant improvement over the Epic model restricted to HIV care appointments (P<.001). Features identified as important in the alternate model included lead time, appointment length, HIV viral load >200 copies per mL, lower CD4 T cell counts (both 50 to <200 cells/mm3 and 200 to <350 cells/mm3), and female sex.

    Conclusions: For both models among people with HIV, performance was significantly lower than reported by Epic. The improvement in the performance of the alternate model over the proprietary Epic model demonstrates that, among people with HIV, the inclusion of demographic information may enhance the prediction of appointment attendance. The alternate model further reveals that the prediction of appointment attendance in people with HIV can be improved by using HIV clinical information such as CD4 count and HIV viral load test results as features in the model.

    J Med Internet Res 2023;25:e43277

    doi:10.2196/43277

    Keywords

    no-show; prediction model; Epic systems; people with HIV; human immunodeficiency virus; electronic medical record; external validation; technology; model; care; patient; HIV 


    Epic Systems is a major provider of health information technology in the United States, providing electronic medical records for more than 250 million patients [1]. Epic’s platform includes predictive models for patient care, including a model that predicts a patient’s probability of being a no-show for an outpatient health care appointment. Epic reports the area under the receiver operating characteristic curve (AUC) of this model to range between 0.71 to 0.81, but the model has not been externally validated in certain groups of patients, including people with HIV [2]. Previous external validation of Epic’s proprietary sepsis prediction model showed reduced performance [3].

    Regular medical care for people with HIV is of the utmost importance [4,5]. Missed medical appointments among people with HIV are independently associated with increased mortality [6]. Additionally, HIV is a disease of extreme disparity, with persons of racial and sexual minorities disproportionately impacted [7]. A no-show predictive model among people with HIV could improve clinical care by allowing providers to proactively engage patients at high risk of missing appointments. We examined the performance of Epic’s no-show model among people with HIV at an academic medical center and whether the performance was impacted by the addition of demographic and HIV clinical information.


    Overview

    On January 21, 2022, the University of Chicago Medicine (UCM) Epic system implemented the second version of the Epic no-show predictive model. The second version of this model is a random forest using 22 features developed using data from 600,000 appointments from February to October 2020 from two health care organizations [2]. The 600,000 appointments used to train the model were not limited to HIV-related appointments. The health care organizations that contributed data for the creation of this model were not identified by Epic [2]. Features included variables related to past appointment history and appointment characteristics. The model output is the predicted probability of each patient being a no-show for each appointment.

    To measure the performance of the Epic model among people with HIV, we used an existing registry of people with HIV at UCM. We then obtained encounter data from all in-person appointments among people with HIV from January 21 to March 30, 2022, and compared the predicted no-show probability at the time of the encounter to the actual outcome of these appointments. We also examined the performance of the Epic model among people with HIV for only HIV care appointments in the infectious diseases department.

    We further compared the no-show model among people with HIV for HIV care appointments to an alternate random forest model we created using a subset of seven readily accessible features used in the Epic model and four additional features related to HIV clinical care or demographics (Table 1). Additional features added to the model included viral load (<200 vs ≥200 copies/mL) and CD4 T cell count (<50, <200, <350, ≥350 cells/mm3) as well as race (Black or African American, White, more than one race, patient declined, and unknown/not reported) and sex at birth (male and female). Features were either continuous (appointment lead time), binary (referral to appointment), or categorical (race).

    For HIV care appointment models, data were randomly split into training (70%) and testing (30%) data sets to standardize the comparison. Model performance was evaluated using AUC via DeLong et al’s [8] method. Feature importance in the model was evaluated using scaled Gini index scores, with higher scaled Gini index scores indicating features with a greater impact on model prediction [9].

    Table 1. No-show prediction model and clinical features for people with HIV at the University of Chicago Medicine, January 21 to March 30, 2022.
    ModelPatients, nRows, nFeatures, nClinical features
    Epic, people with HIV, all appointments model674910022Appointment changed, appointment day of the week, appointment hour of day, appointment month of year, appointment zip code, confirmation status, department ID, department specialty, last communication, appointment lead time, patient called, referral to appointment, referral to appointment required, rescheduled appointment, service area ID, visit type, no-show rate, number of past appointments, number of past canceled appointments, number of past EDa visits, number of past hospitalizations, number of scheduled appointments
    Epic, people with HIV, HIV care appointments model331283522Appointment changed, appointment day of the week, appointment hour of day, appointment month of year, appointment zip code, confirmation status, department ID, department specialty, last communication, appointment lead time, patient called, referral to appointment, referral to appointment required, rescheduled appointment, service area ID, visit type, no-show rate, number of past appointments, number of past canceled appointments, number of past ED visits, number of past hospitalizations, number of scheduled appointments
    Alternate, people with HIV, HIV care appointments model331283511Department ID, department specialty, appointment lead time, referral to appointment, appointment rescheduled, visit type, appointment changed, viral load categories, CD4 categories, race, sex

    aED: emergency department.

    Ethical Considerations

    This project underwent a formal review and received a determination of quality improvement according to UCM institutional policy. Thus, this initiative was not considered human subjects research and was not reviewed by the institutional review board. All UCM patient data were deidentified prior to use.


    We identified 674 people with HIV who contributed 1406 total scheduled in-person appointments during the study period. The performance of the Epic model among people with HIV for all appointments in any outpatient clinic had an AUC of 0.65 (95% CI 0.63-0.66). When we restricted the data to include only HIV care clinic appointments, we identified 331 people with HIV who contributed 440 infectious disease appointments. The AUC of the Epic model for HIV care appointments among people with HIV was 0.63 (95% CI 0.59-0.67), and there was no significant difference in performance compared to the model that included all appointments (P=.36). The alternate model we created for people with HIV attending HIV care appointments had an AUC of 0.78 (95% CI 0.75-0.82), a significant improvement over the Epic model restricted to HIV care appointments (P<.001; Figure 1). Features identified as important in this model included ones from the original Epic model, such as appointment lead time and appointment length. However, some of the new demographic and clinical features were also identified as important, such as viral load >200 copies per mL, lower CD4 T cell counts (both 50 to <200 cells/mm3 and 200 to <350 cells/mm3), and female sex.

    ‎
    Figure 1. No-show prediction model receiver operating characteristic curves for PwH at the University of Chicago Medicine, January 21 to March 30, 2022. (A) Epic PwH all appointments random forest model. (B) Epic PwH HIV care appointments random forest model. (C) Alternate PwH HIV appointments random forest model. AUC: area under the receiver operating characteristic curve; PwH: people with HIV.

    We examined the performance of the Epic no-show predictive model among people with HIV using all appointments regardless of specialty and among HIV care appointments at the infectious disease clinic. For both models among people with HIV, performance was significantly lower than reported by Epic.

    We found that a model that incorporated a subset of the features used in the original Epic model along with demographic and HIV clinical information performed better among people with HIV attending HIV care appointments. Similar to previous studies, our model showed that HIV clinical information (CD4 count and HIV viral load laboratory test results) are important for predicting future appointment attendance in people with HIV [10,11]. The inclusion of demographic factors further improved model performance.

    A limitation of this project was our inability to fully replicate all 22 features from Epic’s no-show prediction model due to the proprietary nature of the model. We were able to identify seven features used by Epic and include these as well as additional clinical and demographic features in the alternate model to see if the additional features would improve the model performance among people with HIV. Two of the model features (race and sex) added in our alternate model were originally part of version 1 of the Epic no-show model but were later removed due to concerns about adverse impacts on marginalized groups [2].

    The improvement in the performance of the alternate model over the proprietary Epic model demonstrates that among populations with extreme disparities, such as people with HIV, the inclusion of demographic information may enhance the prediction of appointment attendance. The alternate model further reveals that the prediction of appointment attendance in people with HIV can be improved by using HIV clinical information such as CD4 count and HIV viral load test results as features in the model. Hospitals and electronic medical record systems interested in developing or improving their no-show prediction models among diseases with high patient disparity may benefit from including these features.

    Acknowledgments

    The authors would like to thank Dr John Fahrenbach for his participation and contributions to this project.

    Data Availability

    The data sets generated or analyzed during this study are not publicly available due to patient privacy but are available from the corresponding author upon reasonable request.

    Conflicts of Interest

    JPR has received consulting fees from Gilead Sciences. EEF has received the City of Chicago Ryan White grant and the FOCUS grant from Gilead Sciences.

    1. About us. Epic.   URL: https://www.epic.com/about [accessed 2022-07-21]
    2. Galaxy - cognitive computing model brief: risk of patient no-show (version 2). Epic. 2022 Feb 28.   URL: https://galaxy.epic.com/?#Browse/page=1!68!50!100020266&from=Galaxy-Redirect [accessed 2022-09-01]
    3. Wong A, Otles E, Donnelly JP, Krumm A, McCullough J, DeTroyer-Cooley O, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 2021 Aug 01;181(8):1065-1070 [FREE Full text] [CrossRef] [Medline]
    4. Li Z, Purcell DW, Sansom SL, Hayes D, Hall HI. Vital signs: HIV transmission along the continuum of care - United States, 2016. MMWR Morb Mortal Wkly Rep 2019 Mar 22;68(11):267-272. [CrossRef] [Medline]
    5. Farnham PG, Gopalappa C, Sansom SL, Hutchinson AB, Brooks JT, Weidle PJ, et al. Updates of lifetime costs of care and quality-of-life estimates for HIV-infected persons in the United States: late versus early diagnosis and entry into care. J Acquir Immune Defic Syndr 2013 Oct 01;64(2):183-189. [CrossRef] [Medline]
    6. Mugavero MJ, Davila JA, Nevin CR, Giordano TP. From access to engagement: measuring retention in outpatient HIV clinical care. AIDS Patient Care STDS 2010 Oct;24(10):607-613 [FREE Full text] [CrossRef] [Medline]
    7. HIV and men. Centers for Disease Control and Prevention. 2022 Jun 28.   URL: https://www.cdc.gov/hiv/group/gender/men/index.html [accessed 2022-07-24]
    8. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988 Sep;44(3):837-845. [Medline]
    9. Breiman L. Random forests. Machine Learning 2001;45:5-32. [CrossRef]
    10. Wawrzyniak AJ, Rodríguez AE, Falcon AE, Chakrabarti A, Parra A, Park J, et al. Association of individual and systemic barriers to optimal medical care in people living with HIV/AIDS in Miami-Dade County. J Acquir Immune Defic Syndr 2015 May 01;69 Suppl 1(0 1):S63-S72 [FREE Full text] [CrossRef] [Medline]
    11. Ridgway JP, Ajith A, Friedman EE, Mugavero MJ, Kitahata MM, Crane HM, et al. Multicenter development and validation of a model for predicting retention in care among people with HIV. AIDS Behav 2022 Oct;26(10):3279-3288. [CrossRef] [Medline]

    ‎
    AUC: area under the receiver operating characteristic curve
    UCM: University of Chicago Medicine

    Edited by T Leung; submitted 06.10.22; peer-reviewed by M Kapsetaki, A Gangadhara Rao; comments to author 30.11.22; revised version received 20.12.22; accepted 26.02.23; published 29.03.23

    Copyright

    ©Joseph A Mason, Eleanor E Friedman, Juan C Rojas, Jessica P Ridgway. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 29.03.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.