We used a retrospective cohort study design with a study period from May 19, 2023, to May 19, 2024. Data were extracted on June 7, 2024. We used the WSIC database, which contains deidentified primary, secondary, tertiary, emergency department, and social care data for over 2.3 million individuals registered with a GP in NWL in 2023 [18].

On May 19, 2023, the model generated hypertension predictions for 1,802,920 individuals, which we accessed as a static dataset. Many of these people would have been included in the original training cohort, given that they had met the inclusion criteria. This was a subset of the patient population in NWL who, at the time, were registered with a GP in NWL, were 16 years or older, and were not already diagnosed with hypertension. Patients with a recorded diagnosis of hypertension and an absent diagnosis date were excluded, as were patients with missing data on gender or prior long-term conditions. Since the predictive model only included male and female genders, we also excluded patients with recorded gender “other.” Full details of how we arrived at the full analytic sample are shown in the Results section.

We extracted 1 subsequent year of this cohort’s primary care medical record history on June 7, 2024, and used these data to evaluate the predictions made by the model. To ascertain each patient’s hypertension status over the study period, we defined a positive diagnosis of hypertension as a patient documented within the Quality and Outcomes Framework (QOF) hypertension register in the WSIC database [23]. Patients were considered to have a negative diagnosis if they had at least one recorded BP reading within the study period but no positive diagnosis recorded by their GP. This captures cases in which an initial BP reading was at least 140/90 mm Hg, but subsequent clinical investigation indicated that the patient did not have hypertension. This is in line with NICE guidelines [24]. If there was no interaction in which a BP measurement was taken and no hypertension diagnosis status documented, we counted the patient’s hypertension status as “not ascertained.” In our main analysis, we excluded individuals whose hypertension status was not ascertained.

We evaluated the model’s predictions against each patient’s known hypertension status, either positive or negative, derived from their GP records as above.

In the development of the model, sensitivity and specificity were computed using 2 probability thresholds, 50% (0.5) and 60% (0.6), with those over the threshold predicted to have hypertension. Specificity and overall performance of the model were best using the 60% threshold. Since the 60% threshold was selected by the consultancy firm for the prediction model, we primarily use it to report the results of the evaluation. Results using the 50% threshold are included in Table S1 in Multimedia Appendix 1.

Our primary method to evaluate the model was computing the sensitivity and specificity of the model-predicted hypertension status against the study-observed hypertension status. Since these statistics are independent of the prevalence, they are unaffected by differences in rates of ascertainment between those with hypertension and those without. We then computed the positive and negative predictive values (NPVs) and their respective CI from the observed sensitivity and specificity, using published national estimates of true hypertension prevalence [25]. We chose this approach because the premise of the model is that there are many cases of undiagnosed hypertension in the population, so labeling all who did not get a positive diagnosis during the study as not having hypertension would introduce significant misclassification bias into the evaluation. We conducted a descriptive analysis of the study population by sociodemographic groups, including gender, age, ethnicity, religion, language, deprivation, and residence in care homes. Using a confusion matrix, which summarizes the performance of predictive algorithms, we assessed the sensitivity and specificity of the model in the population overall [26]. We fitted multivariable logistic regression models to assess variation in sensitivity and specificity of predictions across demographic groups. Patients with missing data on ethnicity or Index of Multiple Deprivation (IMD) quintile were excluded from these regression models.

To understand the extent to which the use of a random forest model conveyed advantages in predictive power over simpler and more transparent regression models, we also fitted 2 logistic regression models, the first including age group, gender, ethnicity, and IMD quintile as predictors, and the second including only age group. We compared the sensitivity and specificity of these models to the random forest model. Further details are provided in Multimedia Appendix 1.

Finally, we conducted 2 sensitivity analyses. First, we reran our analysis while excluding those who died or deregistered from their GP during the study period and who were included in the main analysis. Second, we reran our analysis, classifying all those who did not have a positive diagnosis as not having hypertension. Results of both sensitivity analyses are presented in Tables S2-S8 in Multimedia Appendix 1.

The National Health Service (NHS) Health Research Authority West Midlands—Solihull Research Ethics Committee granted ethical approval covering this study under the WSIC Platform (REC: 23/WM/0196, IRAS: 333128), waiving the need for informed consent or compensation since the study was retrospective and no identifiable patient data were made available for or used in this research. Data access for this study was approved by the NWL Data Access Committee (ID-271‐2). All methods in this study were carried out in accordance with the guidelines and regulations for ethics and study approval.

We conducted an independent pragmatic evaluation of a third-party predictive model, commissioned by NWL integrated care board, using new data not used in training and validation of the model. The model was able to correctly identify 62.7% (10,065/16,061) of cases of undiagnosed hypertension and 60.7% (197,337/325,177) of people without hypertension. Both the sensitivity and specificity of the model were moderate, which means that there will be a large number of missed cases as well as false positives, leading to some people not receiving the help they need and others being called in for a BP check when not necessary, potentially wasting resources.

We also found that the model’s performance varied across demographic groups. The results showed that the model was significantly better at ruling out hypertension in White patients (OR 8.11, 95% CI 7.86-8.37) compared with Asian patients, as demonstrated by the large disparity in specificity between the 2 groups. Furthermore, though sensitivity was high in Black patients (OR 1.72, 95% CI 1.45-2.04), specificity was lowest in this group (OR 0.67, 95% CI 0.64-0.69), which indicates a disproportionate rate of false positives for these patients. The results showing the differences in model performance between demographic groups should be handled with caution, as participants in groups with higher rates of false-positive predictions also tend to have higher rates of misdiagnosis [28].

We found that a simple approach using logistic regression based on demographic variables achieved comparable performance to the random forest model, in terms of AUROC (0.66 vs 0.67, respectively), sensitivity (62.8% vs 62.7%), and specificity (61.0% vs 60.7%). A regression model using only age group achieved higher sensitivity at the cost of lower specificity. Overall, these findings indicate that simple regression models should be considered alongside more complicated machine-learning models. When the transparency afforded by the straightforward interpretation of regression model coefficients is not outweighed by significant increases in predictive power, more transparent models should be preferred.

The results of our sensitivity analysis indicated that, with respect to most variables, our findings were not affected by systematic differences between those who were ascertained during the study period and those who were not. However, there was a difference in the effect of gender on the specificity of the model, indicating that there may be systematic differences in relation to gender between those who were ascertained during the study period and those who were not. The fact that women were more likely to be ascertained than men, in combination with the fact that most people did not get diagnosed with hypertension during the study period, meant that the main analysis excludes more true negatives for men than for women. This is why the specificity is higher in men when including those who were not ascertained as negatives, whereas it is higher in women when using only confirmed negatives in the main analysis. However, given that a proportion of the nonascertained group will, in fact, have hypertension, the finding of higher specificity in men in the sensitivity analysis should be interpreted with caution. The finding that women were more likely to be ascertained than men is consistent with the literature indicating that men are less likely than women to have consulted a GP in the last 12 months and have fewer health care consultations than women after accounting for consultations related to reproductive issues [29-31].

Using national estimates of true prevalence, the PPV, the proportion of correct positive predictions made by the model, ranged from 31.5% (95% CI 31.2-31.8) to 42.9% (95% CI 42.5-43.2). Estimates of the NPV, the proportion of correct negative predictions made by the model, ranged from 77.6% (95% CI 77.2-77.9) to 84.9% (95% CI 84.7-85.2). Overall, the PPV was low, indicating that the model was not reliable in its positive predictions and that there was a high likelihood of false positives. However, the NPV was high, demonstrating that false negatives were less likely and negative predictions were more reliable.

Compared with previous studies that used large-scale EHR data for hypertension risk prediction and case-finding, this study is uniquely positioned to provide a real-world evaluation of model performance across demographic features and the implications for its use in practice [13-16]. Our study was an independent evaluation of a sector-commissioned case-finding model, whereas other studies primarily involved the development and in-house validation of these models. For instance, 1 study developed a model predicting the 2-year risk of hypertension and prospectively validated it in a separate cohort. This model was found to predict hypertension effectively; however, this study only included patients in the desert regions of northwest China, reducing the generalizability of the results to larger, more diverse populations. However, consistent with our study, which found age to be by far the most influential variable in the model, Cheng et al [14] found age to be a significant predictor of the risk of incident hypertension over 2 years.

Another study compared several different algorithms in the development of a hypertension risk prediction model and validated the model in 2 countries, using a Korean training cohort and a Japanese testing cohort [15]. This study also compared the 5-fold cross-validation accuracy across 6 machine-learning models and found that the random forest model performed well in terms of specificity and overall accuracy but had low sensitivity, consistent with the performance of the commissioned model we evaluated [15].

A 2022 systematic review of hypertension prediction machine-learning models found that there is an increasing trend in models to predict hypertension with moderate to high predictive performance. These models used a variety of methods and included a range of features, such as clinical, socioeconomic, and nutritional predictors of hypertension [16].

A 2018 systematic review, which summarized evidence on interventions to improve detection of hypertension in primary care, found that ethnic and socioeconomic differences were not major influencing factors in detection among older age groups, whereas social isolation was an influential factor [32]. Our results, however, showed that the model performed better at identifying cases in older, Black, and more deprived groups, while it was better at ruling out hypertension in younger, White, and less deprived groups. These results are positive with respect to the fact that 1 focus of the national CORE20PLUS5 approach in England is hypertension case-finding in the most deprived 20% of the population [33].

A 2017 systematic review analyzed 107 studies on risk prediction models that used EHR data. These models were developed to infer the clinical status of patients where this was unknown [12]. This review called for improvements in the use of evaluation metrics, such as PPV, which take prevalence into account, to better assess the clinical utility of the developed models. Our study evaluated the PPV and NPV using national estimates of hypertension prevalence to get a range of values that could indicate the reliability of the model’s predictions in a more realistic and clinically useful degree.

This illustrates the broader need for high-quality, independent evaluation of commissioned statistical and machine-learning models, especially where such models are intended for use in clinical care or management. With health systems internationally operating within tightly constrained resources, it is important that analytic solutions using routinely collected data have real-world value and consider health equity.

A strength of our evaluation was that it was an independent, retrospective, real-world, postdeployment evaluation of a predictive model for undiagnosed hypertension in a large population. We used confirmed diagnoses to ascertain positive cases of hypertension rather than proxy measures such as above-threshold BP measurements—an approach that strengthens clinical relevance and bridges the gap between prediction and real-world utility. This evaluation occurred before the results of the model were implemented, ensuring that the study is not measuring the intervention’s yield rather than the independent predictive performance of the case-finding model. Using information about clinical hypertension diagnosis and measurements from GP records combines 2 indicators of hypertension status. This is also a key strength compared to methods common in the literature, which rely only on BP measurements and thresholds to classify true positive and negative cases, or use information about a formal diagnosis and assume that the absence of a diagnosis is equivalent to confirmation of negative hypertension status. This is especially problematic given the premise of significant numbers of undiagnosed cases in the population. The results were robust to excluding those who died or left their GP during the study period, and only the effect of gender changed when including patients whose hypertension status was not ascertained during the study.

A novel aspect of our evaluation is its focus on equity in model performance, which is often overlooked in evaluation studies [20]. This study used the WSIC dataset, which provides high coverage of the demographically diverse NWL population [18]. By investigating the predictive performance of the model across social determinants of health using the WSIC data, our evaluation study aligns with a growing recognition of fairness as a foundational criterion in the assessment of clinical models, not an afterthought [21]. Such equity-focused evaluation remains relatively rare in deployed models, especially in the context of detecting undiagnosed conditions [13-15,22].

A limitation of our approach is that it depends on the accuracy of the coding. We assumed that GPs had followed up with all patients with elevated BP measurements and that there had been no long delay in diagnosing hypertension. Although this is within the framework of clinical decision-making, there may be patients who were not diagnosed with hypertension because of a lack of follow-up or nonadherence to guidelines by the GPs. This would result in some patients being wrongly classified as not having hypertension when, in fact, they did have the condition. We also did not account for time to diagnosis, loss to follow-up, or the competing risk of death. However, our sensitivity analysis, excluding those who died or left the practice for another outside NWL, mitigated this limitation to some extent. Lastly, the scope of this study did not include examination of design decisions made in the model development phase, or issues such as feature importance. Future studies evaluating predictive models should consider including such analyses to provide insights into how the model is working, in addition to how well it is working.

Overall, this evaluation contributes to the emerging field of public health data science by demonstrating how independent evaluation within the same health system can reveal demographic biases and performance variations that may not be apparent during model development. Use of the predictive model has the potential to identify a considerable portion of people living with undiagnosed hypertension, with a view to providing earlier treatment and better health outcomes, although more transparent models are likely to offer similar benefits. Current practice in the NHS is for every individual without preexisting cardiovascular disease, diabetes, or kidney disease to be offered health checks every 5 years from the age of 40 years to 74 years [34]. Use of the model could therefore enable diagnoses earlier in the time periods between health checks. However, a potentially more important group from the perspective of case-finding is younger people who are not offered health checks, and the model demonstrated negligible utility among these groups.

A limitation of the model was the use of area-level proxies in the census data to infer individual clinical risk [35]. Although the census data used was at the most granular level, the ecological fallacy could be a concern when interpreting the model’s predictions. The model was also limited by the available data, including missingness of information in variables including ethnicity, deprivation, language, and religion. As a result, the model may have, to some extent, captured variation in documentation rather than underlying clinical associations.

While the model offers moderate sensitivity and specificity, it is still more likely that a prediction of hypertension is wrong than right. Furthermore, there were wide variations in sensitivity and specificity by demographic groups. Therefore, while the model may be useful as part of a targeted screening program to identify patients likely to benefit from having their BP measured, it should be considered alongside other strategies to ensure certain groups are not disadvantaged, such as enhanced community BP screening and opportunistic measurement during health care visits. For example, given that the model performs poorly in identifying undiagnosed hypertension among younger people, there may be a need for a separate strategy focused on this group.

When integrating the model alongside other strategies, one approach that could be of use would be to deploy the model within strata of the population where it performs well and use alternative strategies where it performs less well. For instance, tables showing descending cumulative sensitivity and specificity are shown in Multimedia Appendix 1 and could be used to identify strata where the model may be of use (Tables S9 and S10 in Multimedia Appendix 1). Alternatively, the model could be deployed across the whole population, with additional strategies supplementing the predictions made for population strata where the model performs poorly.

Several other considerations could be made for the continued and improved use of this model. First, conducting studies to routinely evaluate the model’s predictions over a longer period and the impact of the use of model results on clinical outcomes could greatly improve the understanding of its impact and utility as a practical tool for population health. Second, the findings of this analysis could be incorporated into future cost-benefit analyses to explore the financial impact of such case-finding models on the health system. Finally, collaborating with policymakers to use the findings to inform health care policies and resource allocation, based on analyses like ours and especially by targeting high-risk groups for hypertension screening, could improve the detection and treatment of undiagnosed hypertension, with consequent improvements in health outcomes in NWL and beyond.

The results of this analysis are well-placed to inform deployment of the model alongside other strategies in practice, to drive further enhancements to the model, guide future cost-benefit analyses, steer resource allocation for the detection of undiagnosed hypertension, and contribute to health equity efforts and inform public health policy in NWL. Future research into machine learning techniques that incorporate equity more intentionally is needed to improve the development of such case-finding tools for hypertension. Future work could also include a randomized prospective study to minimize potential misclassification bias due to incomplete patient follow-up or variation in GPs’ application of diagnosis guidelines.

This independent pragmatic evaluation highlights both the value and limitations of predictive modeling for addressing undiagnosed hypertension—a major and persistent public health challenge. The findings of this study show that the model’s predictions for those without hypertension were more reliable than its predictions for those with the condition. The sensitivity and specificity of the model varied across age, ethnicity, and socioeconomic deprivation. Few predictive models in practice are evaluated for bias, real-world performance, or against more parsimonious models, and our findings illustrate why such assessments are critical for equitable and effective implementation. These findings highlight the model’s potential utility in targeting groups for hypertension screening rather than for clinical use with individual patients, emphasizing the need for tailored strategies across different demographic groups to enhance early detection and management of hypertension with a focus on health equity.