This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Asthma affects a large proportion of the population and leads to many hospital encounters involving both hospitalizations and emergency department visits every year. To lower the number of such encounters, many health care systems and health plans deploy predictive models to prospectively identify patients at high risk and offer them care management services for preventive care. However, the previous models do not have sufficient accuracy for serving this purpose well. Embracing the modeling strategy of examining many candidate features, we built a new machine learning model to forecast future asthma hospital encounters of patients with asthma at Intermountain Healthcare, a nonacademic health care system. This model is more accurate than the previously published models. However, it is unclear how well our modeling strategy generalizes to academic health care systems, whose patient composition differs from that of Intermountain Healthcare.
This study aims to evaluate the generalizability of our modeling strategy to the University of Washington Medicine (UWM), an academic health care system.
All adult patients with asthma who visited UWM facilities between 2011 and 2018 served as the patient cohort. We considered 234 candidate features. Through a secondary analysis of 82,888 UWM data instances from 2011 to 2018, we built a machine learning model to forecast asthma hospital encounters of patients with asthma in the subsequent 12 months.
Our UWM model yielded an area under the receiver operating characteristic curve (AUC) of 0.902. When placing the cutoff point for making binary classification at the top 10% (1464/14,644) of patients with asthma with the largest forecasted risk, our UWM model yielded an accuracy of 90.6% (13,268/14,644), a sensitivity of 70.2% (153/218), and a specificity of 90.91% (13,115/14,426).
Our modeling strategy showed excellent generalizability to the UWM, leading to a model with an AUC that is higher than all of the AUCs previously reported in the literature for forecasting asthma hospital encounters. After further optimization, our model could be used to facilitate the efficient and effective allocation of asthma care management resources to improve outcomes.
RR2-10.2196/resprot.5039
In the United States, 7.7% of people have asthma, which causes 188,968 hospitalizations, 1,776,851 emergency department (ED) visits, and 3441 deaths annually [
Owing to its limited service capacity, a care management program normally enrolls at most 3% of patients with a particular condition [
This study evaluates the generalizability of our modeling strategy to the UWM, an academic health care system. Similar to the Intermountain Healthcare model [
This study makes the following 3 innovative contributions:
We conducted the first evaluation of the generalizability of our modeling strategy to an academic health care system.
We evaluated the predictive power of 71 new features, which were not used in our previous study [
We evaluated the generalizability of our Intermountain Healthcare model to the UWM and the generalizability of our UWM model to Intermountain Healthcare. To the best of our knowledge, this is the first study to evaluate model generalizability in both directions. Previously, model generalizability was evaluated solely in one direction by assessing the performance of a model built using one site’s data on another site’s data [
The institutional review boards of the UWM and Intermountain Healthcare approved this secondary analysis study on clinical and administrative data.
The UWM is the largest academic health care system in Washington State. Its enterprise data warehouse contains clinical and administrative data from 3 hospitals and 12 clinics for adults. Our patient cohort covered adult patients with asthma (age ≥18 years) who visited any of these UWM facilities between 2011 and 2018. We defined a patient as having asthma in a specific year if the encounter billing database contained at least one asthma diagnosis code (International Classification of Diseases, Ninth Revision [ICD-9]: 493.0x, 493.1x, 493.8x, 493.9x; International Classification of Diseases, Tenth Revision [ICD-10]: J45.x) record of the patient in that year [
The prediction target was from our previous study [
The time periods used to compute the features and prediction target for an index year and patient pair. PCP: primary care provider.
The UWM enterprise data warehouse supplied a structured data set that contained clinical and administrative data on our patient cohort’s encounters at the 3 UWM hospitals and 12 UWM clinics between 2011 and 2019.
Similar to our previous study [
Every input data instance to the predictive model addresses a unique index year and patient pair and is used to forecast the patient’s outcome in the subsequent 12 months, that is, the 12 months after the end of the index year. For that pair, we computed the patient’s age and primary care provider (PCP) on the last day of the index year. The PCP identified was the patient’s last PCP recorded in the electronic medical record system on or before the last day of the index year. As
Our UWM data set included peak expiratory flow values, which were absent in the Intermountain Healthcare data set adopted in our previous study [
As presented in
Here, TP stands for true positive. TN stands for true negative. FP stands for false positive. FN stands for false negative.
The confusion matrix.
Outcome class | Future asthma hospital encounters | No future asthma hospital encounter |
Forecasted future asthma hospital encounters | True positive | False positive |
Forecasted no future asthma hospital encounter | False negative | True negative |
We performed a 1000-fold bootstrap analysis [
As in our previous paper [
This study mainly evaluated our modeling strategy’s generalizability to the UWM by using the UWM training set to train multiple models and then checking their performance on the UWM test set. In addition, we conducted 2 experiments to evaluate the generalizability of our models across health systems.
In the first experiment, we evaluated the generalizability of our Intermountain Healthcare model to the UWM. Previously, we developed both a simplified model and a full model on the Intermountain Healthcare data set [
In the second experiment, we evaluated the generalizability of our UWM model to Intermountain Healthcare. We used a simplified UWM model, which used only the top features whose importance values calculated by XGBoost on the UWM training set were ≥0.01. For any top feature that was newly introduced in this study and was not used in our previous study [
Each data instance addresses a unique index year and patient pair.
Demographic and clinical characteristics of patients with asthma at the University of Washington Medicine during 2011-2017.
Characteristic | Data instances (N=68,244), n (%) | Data instances connecting to asthma hospital encounters in the subsequent 12 months (n=1184), n (%) | Data instances connecting to no asthma hospital encounter in the subsequent 12 months (n=67,060), n (%) | ||||
|
|||||||
|
<40 | 23,459 (34.38) | 466 (39.36) | 22,993 (34.29) | |||
|
40-65 | 33,889 (49.66) | 583 (49.24) | 33,306 (49.67) | |||
|
>65 | 10,896 (15.97) | 135 (11.40) | 10,761 (16.05) | |||
|
|||||||
|
Male | 24,198 (35.46) | 551 (46.54) | 23,647 (35.26) | |||
|
Female | 44,046 (64.54) | 633 (53.46) | 43,413 (64.74) | |||
|
|||||||
|
American Indian or Alaska native | 1358 (1.99) | 32 (2.70) | 1326 (1.98) | |||
|
Asian | 5721 (8.38) | 96 (8.11) | 5625 (8.39) | |||
|
Black or African American | 8420 (12.34) | 520 (43.92) | 7900 (11.78) | |||
|
Native Hawaiian or other Pacific islander | 673 (0.99) | 14 (1.18) | 659 (0.98) | |||
|
White | 47,747 (69.97) | 507 (42.82) | 47,240 (70.44) | |||
|
Unknown or not reported | 4325 (6.34) | 15 (1.27) | 4310 (6.43) | |||
|
|||||||
|
Hispanic | 3526 (5.17) | 82 (6.93) | 3444 (5.14) | |||
|
Non-Hispanic | 56,309 (82.51) | 1062 (89.70) | 55,247 (82.38) | |||
|
Unknown or not reported | 8409 (12.32) | 40 (3.38) | 8369 (12.48) | |||
|
|||||||
|
Private | 40,009 (58.63) | 424 (35.81) | 39,585 (59.03) | |||
|
Public | 28,787 (42.18) | 756 (63.85) | 28,031 (41.80) | |||
|
Self-paid or charity | 1366 (2.00) | 65 (5.49) | 1301 (1.94) | |||
|
|||||||
|
≤3 | 60,873 (89.20) | 986 (83.28) | 59,887 (89.30) | |||
|
>3 | 7371 (10.80) | 198 (16.72) | 7173 (10.70) | |||
|
|||||||
|
Inhaled corticosteroid | 28,889 (42.33) | 626 (52.88) | 28,263 (42.15) | |||
|
Inhaled corticosteroid and long-acting β-2 agonist combination | 22,015 (32.26) | 499 (42.15) | 21,516 (32.08) | |||
|
Leukotriene modifier | 8171 (11.97) | 201 (16.98) | 7970 (11.88) | |||
|
Long-acting β-2 agonist | 12,293 (18.01) | 374 (31.59) | 11,919 (17.77) | |||
|
Mast cell stabilizer | 47 (0.07) | 4 (0.34) | 43 (0.06) | |||
|
Short-acting inhaled β-2 agonist | 47,808 (70.05) | 1010 (85.30) | 46,798 (69.79) | |||
|
Systemic corticosteroid | 18,699 (27.40) | 614 (51.86) | 18,085 (26.97) | |||
|
|||||||
|
Allergic rhinitis | 11,449 (16.78) | 172 (14.53) | 11,277 (16.82) | |||
|
Anxiety or depression | 19,885 (29.14) | 372 (31.42) | 19,513 (29.10) | |||
|
Bronchopulmonary dysplasia | 1 (0) | 0 (0) | 1 (0) | |||
|
Chronic obstructive pulmonary disease | 3826 (5.61) | 133 (11.23) | 3693 (5.51) | |||
|
Cystic fibrosis | 61 (0.09) | 1 (0.08) | 60 (0.09) | |||
|
Eczema | 3891 (5.70) | 66 (5.57) | 3825 (5.70) | |||
|
Gastroesophageal reflux | 12,291 (18.01) | 238 (20.10) | 12,053 (17.97) | |||
|
Obesity | 7845 (11.50) | 177 (14.95) | 7668 (11.43) | |||
|
Sinusitis | 7261 (10.64) | 89 (7.52) | 7172 (10.69) | |||
|
Sleep apnea | 4556 (6.68) | 88 (7.43) | 4468 (6.66) | |||
|
|||||||
|
Current smoker | 14,081 (20.63) | 255 (21.54) | 13,826 (20.62) | |||
|
Former smoker | 15,530 (22.76) | 221 (18.67) | 15,309 (22.83) | |||
|
Never smoker or unknown | 38,633 (56.61) | 708 (59.80) | 37,925 (56.55) |
Demographic and clinical characteristics of patients with asthma at the University of Washington Medicine in 2018.
Characteristic | Data instances (N=14,644), n (%) | Data instances connecting to asthma hospital encounters in the subsequent 12 months (n=218), n (%) | Data instances connecting to no asthma hospital encounter in the subsequent 12 months (n=14,426), n (%) | ||||
|
|||||||
|
<40 | 4823 (32.9) | 77 (35.3) | 4746 (32.9) | |||
|
40-65 | 6794 (46.4) | 111 (50.9) | 6683 (46.3) | |||
|
>65 | 3027 (20.7) | 30 (13.8) | 2997 (20.8) | |||
|
|||||||
|
Male | 5238 (35.8) | 100 (45.9) | 5138 (35.6) | |||
|
Female | 9406 (64.2) | 118 (54.2) | 9288 (64.4) | |||
|
|||||||
|
American Indian or Alaska native | 281 (1.9) | 8 (3.7) | 273 (1.9) | |||
|
Asian | 1325 (9.1) | 18 (8.7) | 1307 (9.1) | |||
|
Black or African American | 1570 (10.7) | 79 (36.2) | 1491 (10.3) | |||
|
Native Hawaiian or other Pacific islander | 131 (0.9) | 2 (0.9) | 129 (0.9) | |||
|
White | 10,213 (69.7) | 110 (50.5) | 10,103 (70) | |||
|
Unknown or not reported | 1124 (7.7) | 1 (0.5) | 1123 (7.8) | |||
|
|||||||
|
Hispanic | 850 (5.8) | 20 (9.2) | 830 (5.7) | |||
|
Non-Hispanic | 12,566 (85.8) | 196 (89.9) | 12,370 (85.7) | |||
|
Unknown or not reported | 1228 (8.4) | 2 (0.9) | 1226 (8.5) | |||
|
|||||||
|
Private | 10,800 (73.7) | 108 (49.5) | 10,692 (74.1) | |||
|
Public | 8023 (54.8) | 182 (83.5) | 7841 (54.3) | |||
|
Self-paid or charity | 484 (3.3) | 25 (11.5) | 459 (3.2) | |||
|
|||||||
|
≤3 | 10,566 (72.1) | 124 (56.9) | 10,442 (72.4) | |||
|
>3 | 4078 (27.8) | 94 (43.1) | 3984 (27.6) | |||
|
|||||||
|
Inhaled corticosteroid | 6177 (42.2) | 108 (49.5) | 6069 (42.1) | |||
|
Inhaled corticosteroid and long-acting β-2 agonist combination | 4508 (30.8) | 83 (38.1) | 4425 (30.7) | |||
|
Leukotriene modifier | 2176 (14.9) | 46 (21.1) | 2130 (14.77) | |||
|
Long-acting β-2 agonist | 2518 (17.2) | 62 (28.4) | 2456 (17.02) | |||
|
Mast cell stabilizer | 14 (0.1) | 1 (0.5) | 13 (0.09) | |||
|
Short-acting inhaled β-2 agonist | 9704 (66.3) | 164 (75.2) | 9540 (66.1) | |||
|
Systemic corticosteroid | 4163 (28.4) | 120 (55.1) | 4043 (28) | |||
|
|||||||
|
Allergic rhinitis | 2095 (14.3) | 26 (11.9) | 2069 (14.3) | |||
|
Anxiety or depression | 4346 (29.7) | 62 (28.4) | 4284 (29.7) | |||
|
Bronchopulmonary dysplasia | 4 (0) | 0 (0) | 4 (0) | |||
|
Chronic obstructive pulmonary disease | 932 (6.4) | 30 (13.8) | 902 (6.2) | |||
|
Cystic fibrosis | 17 (0.1) | 0 (0) | 17 (0.1) | |||
|
Eczema | 743 (5.1) | 11 (5.1) | 732 (5.1) | |||
|
Gastroesophageal reflux | 2657 (18.1) | 46 (21.1) | 2611 (18.1) | |||
|
Obesity | 1604 (10.9) | 25 (11.5) | 1579 (10.9) | |||
|
Sinusitis | 1372 (9.4) | 15 (6.9) | 1357 (9.4) | |||
|
Sleep apnea | 1499 (10.2) | 24 (11.0) | 1475 (10.2) | |||
|
|||||||
|
Current smoker | 3242 (22.1) | 49 (22.5) | 3193 (22.1) | |||
|
Former smoker | 3494 (23.9) | 41 (18.8) | 3453 (23.9) | |||
|
Never smoker or unknown | 7908 (54.0) | 128 (58.7) | 7780 (53.9) |
As the Chi-square 2-sample test showed, for both the 2011-2017 and 2018 data, the data instances connecting to future asthma hospital encounters and those connecting to no future asthma hospital encounter exhibited the same distribution for anxiety or depression occurrence (
As the Cochran-Armitage trend test [
The number of patients with asthma and their number of visits in each year between 2011 and 2018.
Year | Number of patients with asthma | Number of visits by patients with asthma |
2011 | 6852 | 32,910 |
2012 | 7768 | 40,730 |
2013 | 7754 | 39,385 |
2014 | 9785 | 58,953 |
2015 | 10,587 | 69,285 |
2016 | 12,072 | 78,605 |
2017 | 13,426 | 87,403 |
2018 | 14,644 | 94,875 |
Our automatic machine learning model selection method [
On the UWM test set, our final model yielded an AUC of 0.902 (95% CI 0.879-0.924).
Several features, such as a family history of asthma, were calculated on 2 or more years of data. When we dropped these features and checked solely those features calculated on 1 year of data, the AUC of the model decreased from 0.902 to 0.899. If we used only the top 17 features in Table S2 of
The receiver operating characteristic curve of our final University of Washington Medicine model.
Our final UWM model’s performance measures when the cutoff point for making binary classification was placed at different top percentages of patients with asthma with the largest forecasted risk.
Top percentage of patients with asthma with the largest forecasted risk (%) | Accuracy (N=14,644), n (%) | Sensitivity (N=218), n (%) | Specificity (N=14,426), n (%) | PPVa | NPVb | ||
|
|
|
|
n (%) | N | n (%) | N |
1 | 14,410 (98.4) | 65 (29.8) | 14,345 (99.4) | 65 (44.5) | 146 | 14,345 (98.9) | 14,498 |
2 | 14,316 (97.8) | 91 (41.7) | 14,225 (98.6) | 91 (31.2) | 292 | 14,225 (99.1) | 14,352 |
3 | 14,193 (96.9) | 103 (47.3) | 14,090 (97.7) | 103 (23.5) | 439 | 14,090 (99.2) | 14,205 |
4 | 14,061 (96) | 110 (50.5) | 13,951 (96.7) | 110 (18.8) | 585 | 13,951 (99.2) | 14,059 |
5 | 13,936 (95.2) | 121 (55.5) | 13,815 (95.8) | 121 (16.5) | 732 | 13,815 (99.3) | 13,912 |
6 | 13,806 (94.3) | 129 (59.2) | 13,677 (94.8) | 129 (14.7) | 878 | 13,677 (99.3) | 13,766 |
7 | 13,667 (93.3) | 133 (61) | 13,534 (93.8) | 133 (13) | 1025 | 13,534 (99.4) | 13,619 |
8 | 13,529 (92.4) | 137 (62.8) | 13,392 (92.8) | 137 (11.7) | 1171 | 13,392 (99.4) | 13,473 |
9 | 13,411 (91.6) | 151 (69.3) | 13,260 (91.9) | 151 (11.5) | 1317 | 13,260 (99.5) | 13,327 |
10 | 13,268 (90.6) | 153 (70.2) | 13,115 (90.9) | 153 (10.5) | 1464 | 13,115 (99.5) | 13,180 |
15 | 12,576 (85.9) | 173 (79.4) | 12,403 (86) | 173 (7.9) | 2196 | 12,403 (99.6) | 12,448 |
20 | 11,860 (81) | 181 (83) | 11,679 (81) | 181 (6.2) | 2928 | 11,679 (99.7) | 11,716 |
25 | 11,147 (76.1) | 191 (87.6) | 10,956 (75.9) | 191 (5.2) | 3661 | 10,956 (99.7) | 10,983 |
aPPV: positive predictive value.
bNPV: negative predictive value.
The confusion matrix of our final University of Washington Medicine model when the cutoff point for making binary classification was placed at the top 10% (1464/14,644) of patients with asthma with the largest forecasted risk.
Outcome class | Future asthma hospital encounter, n | No future asthma hospital encounter, n |
Forecasted future asthma hospital encounters | 153 | 1311 |
Forecasted no future asthma hospital encounter | 65 | 13,115 |
For our original simplified Intermountain Healthcare model trained on the Intermountain Healthcare training set [
After we used the UWM training set to retrain our simplified Intermountain Healthcare model [
Our simplified UWM model used only the top 17 features with importance values of ≥0.01. For our simplified UWM model trained on the UWM training set, when we did not retrain the model and applied the model directly to the Intermountain Healthcare test set, the model yielded an AUC of 0.814 (95% CI 0.798-0.830). When we placed the cutoff point for making binary classification at the top 10% (1926/19,256) of patients with asthma with the largest forecasted risk, the model yielded an accuracy of 89.76% (17,285/19,256; 95% CI 89.32-90.18), a sensitivity of 47.2% (383/812; 95% CI 43.8-50.6), a specificity of 91.64% (16,902/18,444; 95% CI 91.24-92.03), a PPV of 19.90% (383/1925; 95% CI 18.16-21.60), and an NPV of 97.52% (16,902/17,331; 95% CI 97.28-97.75).
After we used the Intermountain Healthcare training set to retrain our simplified UWM model, the retrained model yielded on the Intermountain Healthcare test set an AUC of 0.846 (95% CI 0.831-0.859). When we placed the cutoff point for making binary classification at the top 10% (1926/19,256) of patients with asthma with the largest forecasted risk, the model yielded an accuracy of 90.11% (17,351/19,256; 95% CI 89.64-90.56), a sensitivity of 51.2% (416/812; 95% CI 47.6-54.5), a specificity of 91.82% (16,935/18,444; 95% CI 91.43-92.21), a PPV of 21.62% (416/1,925; 95% CI 19.81-23.41), and an NPV of 97.72% (16,935/17,331; 95% CI 97.48-97.93).
We built a model on UWM data to forecast asthma hospital encounters of patients with asthma in the subsequent 12 months.
In Table S2 of
We checked 234 candidate features. Our final UWM model used 30.3% (71/234) of them. Despite being correlated with the outcome, many unused features had no extra predictive power on the UWM data set over the features adopted in our final UWM model.
For our original simplified Intermountain Healthcare model trained on the Intermountain Healthcare training set [
Compared with our full UWM model using 71 features, our simplified UWM model retained nearly all of its predictive power. For our simplified UWM model trained on the UWM training set, when we did not retrain the model on the Intermountain Healthcare data and directly applied the model, the model yielded an AUC of 0.814 on the Intermountain Healthcare test set. This AUC is 0.045 lower than our full Intermountain Healthcare model’s AUC but is still larger than the previously reported AUC of every existing model developed by others for forecasting future hospitalizations and ED visits of patients with asthma (
A comparison of our final University of Washington Medicine model and several existing models for forecasting future hospitalizations and emergency department (ED) visits of patients with asthma.
Model | Prediction target | Number of data instances | Number of features the model adopted | Classification algorithm | Sensitivity (%) | Specificity (%) | PPVa (%) | NPVb (%) | AUCc |
Our final UWM model | Asthma hospital encounters | 82,888 | 71 | XGBoostd | 70.2 | 90.91 | 10.45 | 99.51 | 0.902 |
Our Intermountain Healthcare model [ |
Asthma hospital encounters | 334,564 | 142 | XGBoost | 53.69 | 91.93 | 22.65 | 97.83 | 0.859 |
Loymans et al [ |
Asthma exacerbation | 611 | 7 | Logistic regression | —e | — | — | — | 0.8 |
Schatz et al [ |
Asthma-induced hospitalization in children | 4197 | 5 | Logistic regression | 43.9 | 89.8 | 5.6 | 99.1 | 0.781 |
Schatz et al [ |
Asthma-induced hospitalization in adults | 6904 | 3 | Logistic regression | 44.9 | 87 | 3.9 | 99.3 | 0.712 |
Eisner et al [ |
Asthma-induced hospitalization | 2858 | 1 | Logistic regression | — | — | — | — | 0.689 |
Eisner et al [ |
Asthma-induced ED visit | 2415 | 3 | Logistic regression | — | — | — | — | 0.751 |
Sato et al [ |
Severe asthma exacerbation | 78 | 3 | Classification and regression tree | — | — | — | — | 0.625 |
Miller et al [ |
Asthma hospital encounters | 2821 | 17 | Logistic regression | — | — | — | — | 0.81 |
Yurk et al [ |
Lost day or hospital encounters for asthma | 4888 | 11 | Logistic regression | 77 | 63 | 82 | 56 | 0.78 |
Lieu et al [ |
Asthma-induced hospitalization | 16,520 | 7 | Proportional-hazards regression | — | — | — | — | 0.79 |
Lieu et al [ |
Asthma-induced ED visit | 16,520 | 7 | Proportional-hazards regression | — | — | — | — | 0.69 |
Lieu et al [ |
Asthma hospital encounters | 7141 | 4 | Classification and regression tree | 49 | 83.6 | 18.5 | — | — |
Schatz et al [ |
Asthma hospital encounters | 14,893 | 4 | Logistic regression | 25.4 | 92 | 22 | 93.2 | 0.614 |
Forno et al [ |
Severe asthma exacerbation | 615 | 17 | Scoring | — | — | — | — | 0.75 |
Xiang et al [ |
Asthma exacerbation | 31,433 | — | Recurrent neural network | — | — | — | — | 0.70 |
aPPV: positive predictive value.
bNPV: negative predictive value.
cAUC: area under the receiver operating characteristic curve.
dXGBoost: extreme gradient boosting.
eThe initial paper showing the model did not give the performance measure.
Researchers have built multiple models to forecast future hospitalizations and ED visits of patients with asthma [
It is important to consider the prevalence of the outcome of interest when comparing the performance of different predictive models. Compared with other existing models, the model by Yurk et al [
The recurrent neural network model by Xiang et al [
Excluding the model by Yurk et al [
The prevalence rate of targeted poor outcomes greatly impacts the PPV of any predictive model [
Our final UWM model and our Intermountain Healthcare model [
Differing models in
Our final UWM model has an AUC that is higher than all of the AUCs previously reported in the literature for forecasting asthma hospital encounters, but still had a seemingly low PPV of 10.4% (153/1464). Nevertheless, this model could be valuable in clinical care. First, health care systems such as UWM, Intermountain Healthcare, and Kaiser Permanente Northern California [
Second, as explained earlier, even an impeccable model in theory would reach a low PPV because the poor outcome of interest has a low prevalence rate in our data set. For such an outcome, sensitivity better reflects the model’s potential clinical value than PPV. Our final UWM model had a higher sensitivity than the previously reported sensitivity of every existing model using a comparable prediction target. It is important to note that while asthma hospital encounters have an overall low prevalence rate in the population of patients with asthma, they have significant financial and clinical impacts at both the population and individual patient levels.
Third, a PPV of 10.45% (153/1464) is useful for identifying high-risk patients with asthma to receive low-cost preventive interventions. The following are 4 examples of such interventions: training the patient to record a diary about environmental triggers, coaching the patient to use an asthma inhaler correctly, coaching the patient to use a peak flow meter correctly and giving it to the patient to self-monitor symptoms at home, and asking a nurse to do extra follow-up phone calls with the patient or the patient’s caregiver. These interventions could have a significant impact on patient outcomes.
The final UWM model used 71 features. Reducing the number of features could ease the clinical deployment of our model. To this end, if a minor decrease in prediction accuracy could be tolerated, one could adopt the top few features whose importance values are greater than a given threshold, such as 0.01, and drop the other features. The importance value of a feature varies across health care systems. Ideally, the importance values of the features should first be calculated on a data set from the target health care system before choosing the features to retain.
As is typical with complex machine learning models, an XGBoost model using many features is difficult to interpret. This can limit clinical understandability and adoption, particularly by clinicians who are resistant to using automated tools. In the future, we plan to adopt our previously developed method [
The final UWM model was constructed using XGBoost [
This study has at least 4 limitations that could be interesting topics for future work, as follows:
It is possible to further increase the model accuracy by using features other than those checked in this study. For example, features derived from environmental and physiological data gathered by intelligent wearable devices can have this potential.
This study used purely structured data and checked only nondeep learning classification algorithms. It is possible to further increase the model accuracy by using deep learning as well as features derived from unstructured clinical notes using natural language processing techniques [
Our UWM data set contained no data on patients’ health care use outside of UWM. Therefore, we limited the prediction target to asthma hospital encounters at UWM instead of asthma hospital encounters anywhere. In addition, the features we checked were derived from patients’ incomplete administrative and clinical data [
This study evaluated the generalizability of our modeling strategy to an academic health care system on a single outcome of a complex chronic disease. We recently showed that our modeling strategy also generalizes well to Kaiser Permanente Southern California for the same predictive modeling problem [
In the first evaluation of its generalizability to an academic health care system, our modeling strategy of examining many candidate features to enhance prediction accuracy showed excellent generalizability to the UWM and led to a model with an AUC that is higher than all of the AUCs previously reported in the literature for forecasting asthma hospital encounters. After further optimization, our UWM model could be used to facilitate the efficient and effective allocation of asthma care management resources to improve outcomes.
The list of candidate features considered in this study.
area under the receiver operating characteristic curve
emergency department
false negative
false positive
International Classification of Diseases, Tenth Revision
International Classification of Diseases, Ninth Revision
negative predictive value
primary care provider
positive predictive value
true negative
true positive
University of Washington Medicine
Waikato Environment for Knowledge Analysis
extreme gradient boosting
The authors would like to thank Katy Atwood for helping with the retrieval of the UWM data set and Michael D Johnson for useful discussions. GL, SM, and GD were partially supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health under award R01HL142503. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. YT did the work at the University of Washington when she was a visiting PhD student.
YT participated in data analysis and the writing of the first draft of the paper. GL conceptualized and designed the study, performed a literature review, participated in data analysis, and rewrote the whole paper. AM, AW, SM, GD, and PS provided feedback on various medical issues, contributed to conceptualizing the presentation, and revised the paper.
None declared.