This retrospective observational prediction model development and internal validation study used real-world data from the “Easy Test Know” online HIVST platform. Demographic, transaction, and behavioral data were collected from November 2023 to February 2025. The study was conducted through a single digital service platform serving users across China; because recruitment and data collection occurred online, no physical recruitment centers were involved. The platform integrates a structured HIV risk assessment questionnaire (Multimedia Appendix 1). Based on questionnaire responses, users were classified into low-, medium-, or high-risk categories.

Recruitment occurred through the platform’s routine service pathway rather than a study-specific campaign. Users voluntarily accessed the service, and no additional participants were recruited, contacted, or assigned to an exposure by the research team. Records generated during routine use were screened against prespecified eligibility criteria for this retrospective secondary analysis.

The study population consisted of MSM who completed the platform-based HIV risk assessment and purchased HIVST kits. Participants were included if they were classified as medium or high risk, completed the HIVST pathway, and had an available, usable HIV test result recorded in the platform database. Purchasers without an available or usable test result were excluded rather than being classified as HIV-negative. Low-risk users generally did not proceed to kit purchase and therefore lacked an observed testing outcome within the platform; consequently, they could not be included in outcome-based predictive analyses. MSM eligibility was based on reported male-to-male sexual behavior rather than self-identified sexual orientation.

The platform employs a rule-based HIV risk scoring algorithm (Multimedia Appendix 2) that assigns weighted scores based on demographic characteristics and HIV-related behavioral factors. Users are subsequently categorized into low-, medium-, or high-risk groups according to their final scores. The deployed platform calculated the dimension 1 and 2 base scores and applied its configured dimension 3 combination rules before storing the final continuous score and categorical output. The present study used these stored final outputs without modifying or recalculating the platform’s algorithm. The reconstructed Firth regression models were developed independently of the individual behavioral variables and did not reapply the platform’s dimension 3 rules.

The study used Firth logistic regression to address potential biases that may arise in standard logistic regression. Because HIV-positive outcomes were relatively rare in the study population, Firth logistic regression was used to reduce rare-event bias in parameter estimation and to provide finite estimates under complete or quasi-complete separation. This was relevant because only 58 HIV-positive outcomes were observed, and no HIV-positive outcomes occurred among participants reporting pre-exposure prophylaxis (PrEP) use. The fully adjusted model contained 24 predictor parameters (events-per-variable ratio, 2.42). Firth penalization was therefore used to support finite, less biased coefficient estimation in this rare-event setting.

The platform-based questionnaire collected information on HIV-related demographic and behavioral factors, including sexual orientation, condom use, HIV-positive partner status, number of sexual partners, commercial sex, history of sexually transmitted infections, use of new psychoactive drugs, sexual role, and group sex. The analytical dataset also contained recorded use of PrEP and postexposure prophylaxis (PEP), which were analyzed as separate variables. PEP and PrEP were recorded separately and were not components of the platform’s original scoring algorithm.

Sexual orientation was analyzed as a demographic characteristic distinct from the behavioral definition of MSM. The drug-use item asked whether participants had used new psychoactive drugs but did not capture specific substances, frequency, dose, or recency. The original partner-number response options were 2 to 5, 6 to 9, and greater than 10; the questionnaire did not provide categories for zero or one partner and left exactly 10 partners unspecified.

High-risk exposure was assessed based on 6 predefined location categories, including mobile social networking applications, online dating platforms, parks, bathhouses, public toilets, and bars/clubs. Because participants could report exposure to multiple venues, we constructed a composite indicator termed the high-risk venue diversity score, defined as the number of different high-risk venues reported by each participant (range: 0‐6). This indicator was intended to reflect cumulative exposure across multiple high-risk contexts [19,20]. “Only regular sexual partners” represented zero exposure and was not counted as an additional venue. Although “sex work venues” appeared in the questionnaire, no participant selected this option in the analytical dataset. This derived measure was distinct from the platform’s original weighted scoring algorithm.

The primary outcome was HIV-positive status, defined as a participant-reported positive HIVST result submitted through the platform or a positive result from offline HIV testing reported after completion of the platform-based risk assessment. The outcome was taken directly from the recorded participant report, and no separate blinded outcome adjudication was performed. Participants without a usable returned result were excluded from the analytical dataset; therefore, the observed positivity proportion should not be interpreted as HIV prevalence among all platform purchasers or the broader MSM population.

The study size was determined pragmatically by all eligible records available in the supplied routine-service extract during the prespecified study period; no a priori sample size calculation was performed. Of 10,058 records screened, 87 records from female users were excluded, leaving 9971 male users meeting the platform-based MSM definition. Ten users without a usable self-reported HIVST or offline HIV test result were then excluded, resulting in the analytical sample of 9961; the usable-result return rate in this MSM pathway was therefore 99.9% (9961/9971), and 0.1% (10/9971) lacked a usable result. There were no missing values for the outcome or candidate model predictors in the analytical dataset; all model analyses used the same complete dataset, and no imputation was performed.

Categorical variables are presented as counts and percentages. Comparisons between medium- and high-risk groups and between the observed HIV-negative and HIV-positive groups were performed using chi-square or Fisher exact tests, as appropriate. The complete high-risk venue diversity score (0-6) was additionally evaluated for linear trend using Firth logistic regression, with the score treated as an ordinal continuous variable.

To evaluate HIV risk prediction performance, we constructed and compared three model specifications: (1) a baseline model that contained only the continuous platform risk scores; (2) an enhanced behavioral model that additionally incorporated the high-risk venue diversity scores; and (3) a fully adjusted behavioral model that incorporated additional HIV-related behavioral variables.

Because several behavioral variables had already been incorporated into the platform’s original scoring algorithm, the reconstructed models primarily evaluated whether data-driven weighting of behavioral information could improve predictive performance rather than the addition of entirely new predictors. The platform’s deployed categorical risk output was evaluated separately as an operational comparator.

The continuous platform score was modeled as a linear term. Age and all questionnaire-derived predictors were modeled using the predefined categories described in the Result section and Multimedia Appendix 2. In the reconstructed models, high-risk venue diversity was entered as 0, 1, or ≥2 venues; in a separate trend analysis, the complete score (0-6) was entered as an ordinal continuous variable. The same 9961 complete records were used in all model analyses.

Model discrimination was evaluated using the area under the receiver operating characteristic curve (AUC), and DeLong tests were used to compare AUCs between models. Stratified bootstrap resampling (2000 replicates) was used to estimate 95% CIs for apparent AUCs. Internal validation of the enhanced and fully adjusted Firth models used stratified bootstrap optimism correction. In each resample, the complete model was refitted in the bootstrap sample and evaluated in both the bootstrap sample and the original data; mean optimism was calculated as bootstrap-sample performance minus original-sample performance and subtracted from the apparent estimate. Sampling continued until 1000 successful refits were obtained for each model, and unsuccessful attempts were recorded. Optimism correction was applied to the AUC, calibration-in-the-large, calibration slope, expected-to-observed (E/O) ratio on the log scale, and Brier score. Hosmer-Lemeshow statistics were reported only for apparent calibration. Calibration plots display apparent and bootstrap bias-corrected curves based on 1000 resamples. The number of outcome events per estimated predictor parameter was calculated for each reconstructed model. All analyses were conducted using R software (version 4.5.2; R Foundation for Statistical Computing), and a 2-sided P value <.05 was considered statistically significant.

This study was approved by the Ethics Review Committee of the National Center for AIDS/Sexually Transmitted Disease Control and Prevention, Chinese Center for Disease Control and Prevention (approval number X131022302). The analytical dataset was deidentified before analysis and was accessible only to authorized study personnel; privacy and confidentiality were maintained throughout data handling and analysis. Participants received no additional compensation specifically for this retrospective secondary analysis. All study procedures were conducted in accordance with relevant guidelines and regulations.

Of the 10,058 records in the supplied routine-service extract, 87 records from female users and 10 records from male users without a usable HIV test result were excluded. Among 9971 male users meeting the platform-based MSM definition, 9961 (99.9%) had a usable returned result. The final analysis included these 9961 MSM (Table 1), including 917 (9.2%) classified as medium risk and 9044 (90.8%) classified as high risk by the platform algorithm. Of these participants, 58 had an observed HIV-positive result and 9903 had an observed HIV-negative result.

Compared with participants in the medium-risk group, those classified as high risk differed across multiple behavioral characteristics, including sexual role, high-risk venue diversity, condom use, HIV-positive partner status, number of sexual partners, commercial sex, group sex, drug use, and PEP use (Table 1). These differences describe how the rule-based platform score stratified behavioral profiles.

The distribution of high-risk venue exposure differed substantially between groups. Participants in the high-risk group were considerably more likely to report exposure to 2 or more high-risk venues compared with those in the medium-risk group (6909/9044, 76.4% vs 301/917, 32.8%; P<.001), indicating substantial differences in cumulative behavioral exposure patterns between the 2 risk categories.

Among the 9961 participants, 58 had an observed positive HIV test result, corresponding to an observed positivity proportion of 0.58% (Table 2). Participants classified as high risk had a numerically higher positivity proportion than those classified as medium risk (54/9044, 0.6% vs 4/917, 0.4%), but the difference was not statistically significant. The observed positivity proportion reflects only participants with available test results and should not be interpreted as HIV prevalence among all platform users or the broader MSM population.

The continuous platform risk score showed modest discrimination for observed HIV positivity (AUC 0.646, 95% CI 0.567‐0.725; Table S1 in Multimedia Appendix 3). When the deployed categorical risk output was evaluated directly, its AUC was 0.512 (95% CI 0.477‐0.539). The categorical output achieved high sensitivity (93.1%), classifying 54 of 58 HIV-positive participants as high risk, although its specificity was 9.2% (913/9903); the remaining 4 HIV-positive participants were classified as medium risk.

Observed HIV positivity increased across the well-populated levels of the high-risk venue diversity score: 0% (0/3) at score 0, 0.2% (6/2748) at score 1, 0.5% (18/3824) at score 2, 0.8% (21/2762) at score 3, and 2.1% (13/623) at score 4; only one participant had a score of 5, and none had a score of 6 (Table S4 in Multimedia Appendix 3). In Firth logistic regression, each 1-category increase in the diversity score was associated with higher odds of observed HIV positivity (odds ratio [OR] 2.04, 95% CI 1.53‐2.74; P for trend <.001).

HIV-positive partner status was strongly associated with observed HIV positivity in the fully adjusted model (Table 3), although estimates were imprecise for the very small “yes” group. Compared with participants who were uncertain about their partner’s HIV status, the adjusted OR was 6.57 (95% CI 2.87‐13.58) for those reporting no HIV-positive partner and 172.22 (95% CI 7.76‐27,046.77) for those reporting an HIV-positive partner. PrEP use was inversely associated with observed HIV positivity: none of the 920 PrEP users had a positive result, compared with 58 positive results among 9041 nonusers (adjusted OR 0.10, 95% CI 0.001‐0.68; P=.01). This complete separation was one rationale for the use of Firth regression and should not be interpreted as a causal estimate of PrEP effectiveness.

This study evaluated the performance of a digital HIV behavioral risk stratification system integrated with online HIVST services among MSM with available test results in the medium- and high-risk pathways. The deployed categorical output showed high sensitivity but low specificity, whereas the continuous platform score provided modest discrimination. Outcome-driven behavioral models achieved higher apparent AUCs and retained improved AUCs after bootstrap optimism correction, and high-risk venue diversity was associated with observed HIV positivity. The relatively low observed positivity proportion (58/9961, 0.58%) may partly reflect the platform’s broad, prevention-oriented case-finding strategy, which prioritized sensitivity (54/58, 93.1%) over specificity (913/9903, 9.2%) and therefore included many HIV-negative users rather than restricting testing to a narrowly selected clinical population. These findings address the study aims but apply only to the analyzed population and should not be interpreted as population prevalence or performance across the full risk spectrum.

The categorical and continuous platform results require separate interpretation. Positivity was 0.6% (54/58) in the high-risk group and 0.4% (4/914) in the medium-risk group, but the difference was not statistically significant. This nonsignificant difference between categories does not establish that the platform lacks practical value; rather, it supports evaluating and refining its thresholds using observed outcome data.

The observed association between exposure to multiple high-risk venues and HIV positivity suggests that cumulative venue-based behavioral exposure may provide additional information beyond conventional individual behavioral indicators. These findings support the potential value of integrating multidimensional behavioral exposure measures into digital HIV risk assessment frameworks.

The association between high-risk venue diversity score and HIV positivity may reflect broader patterns of cumulative behavioral exposure. On the one hand, different types of high-risk sites may offer greater anonymity, more frequent temporary partner contacts, and environments where continuous risk prevention is difficult to maintain. Exposure to multiple sites often places individuals in more complex and overlapping high-risk sexual networks. On the other hand, the number of different sites individuals visit may reflect several risk dimensions, such as the frequency of behaviors, sexual behavior diversity, and the degree of social network overlap. Compared with a single behavioral indicator, this metric provides a more comprehensive reflection of cumulative risk intensity, which may help explain the dose-response relationship for HIV positivity [18,20,21]. Because these mechanisms were not measured directly, the venue diversity finding should be interpreted as predictive rather than causal. Nevertheless, it suggests that venue-based exposure measures may contribute additional information to digital HIV risk assessment beyond conventional single-behavior indicators [14,15].

Although drug use was more common among participants classified as high risk by the platform algorithm, it was not significantly associated with observed HIV positivity. The questionnaire item combined new psychoactive drugs but did not capture specific substances, frequency, dose, or recency. Accordingly, the high reported proportion should not be generalized to the broader MSM population, and the null association may reflect limited measurement specificity or statistical power rather than the absence of a relationship with HIV risk.

One possible explanation is that drug use may reflect the clustering of multiple co-occurring high-risk behaviors within the platform’s scoring framework. However, after adjustment for other behavioral factors, its additional predictive contribution appeared limited. These findings suggest that the relative importance of specific behavioral indicators may differ between expert-defined scoring systems and data-driven predictive models.

Compared with many previous studies focusing on individual behavioral factors or offline testing populations [22], the present study evaluated behavioral risk prediction within a real-world online HIVST platform. The observed association between exposure to multiple high-risk venues and HIV positivity is broadly consistent with previous studies suggesting that repeated exposure across diverse sexual or social environments may be associated with increased HIV risk.

In addition, the use of real-world platform-based behavioral and transaction data provides insight into HIVST users who may be less engaged with conventional in-person testing services. Recent digitally supported HIVST programs have also shown that online self-testing can be connected to counseling, prevention, and linkage to care [8,9]. The present study complements this literature by evaluating the correspondence between a prepurchase behavioral assessment and available testing outcomes.

Importantly, the improved performance observed in the reconstructed behavioral models should not be interpreted simply as the addition of entirely new predictors. Several behavioral variables included in the optimized models were already incorporated into the platform’s original rule-based scoring system. Rather, the observed improvement likely reflects differences in how behavioral information was aggregated and weighted. The platform algorithm relies on predefined, expert-assigned weights, whereas the present study applied a data-driven regression framework to estimate the relative contributions of behavioral variables directly from observed outcome data. These findings suggest that optimizing weighting strategies and behavioral feature integration may substantially improve the predictive performance of digital HIV risk stratification tools. It should be noted that the comparison is between a model optimized on the current dataset and a fixed algorithm; the observed improvement partly reflects the statistical advantage of in-sample optimization and does not imply that the platform algorithm is deficient.

From an application standpoint, the reconstructed behavioral models demonstrated improved discrimination compared with the continuous platform score, and the fully adjusted model retained the highest AUC after bootstrap optimism correction (0.713). The optimism-corrected calibration slopes also remained close to 1, supporting the potential value of integrating multidimensional behavioral indicators and outcome-informed weighting into digital HIV risk stratification. These models may provide a useful foundation for refining risk assessment and identifying individuals who could benefit from prioritized follow-up testing, online counseling, self-test support, or linkage to PrEP and other prevention services, particularly in resource-limited settings [8,14,16,23,24]. In a national-level implementation study in Vietnam [9], an online HIVST distribution program successfully linked 97.5% of individuals with reactive results to treatment, while approximately 20% of those with negative results initiated or continued PrEP. This experience further illustrates how appropriately validated digital risk assessment tools could be integrated into the HIV detection, prevention, and treatment continuum. Further external validation of the present models would help confirm their performance and inform appropriate implementation thresholds.

This study has several limitations. First, outcome ascertainment depended on a usable self-test result returned through the platform or an offline result reported after assessment. Participants without usable returned results were excluded rather than classified as HIV-negative, but selective result reporting may still affect the observed positivity proportion and model coefficients. The findings should therefore not be interpreted as HIV prevalence among all purchasers or the broader MSM population.

Second, low-risk users were not represented because they generally did not proceed through the same purchase and result-return pathway. Restricting the analysis to medium- and high-risk participants compressed the evaluated score range; consequently, the reported platform performance applies only to the observed risk spectrum and not to all platform users.

Third, only 58 HIV-positive outcomes were available, and the fully adjusted model included 24 predictor parameters, corresponding to 2.42 events per parameter. Firth penalization reduced coefficient bias and provided finite estimates under separation, but it did not eliminate overfitting. Bootstrap internal validation yielded more conservative AUC estimates, while the corrected calibration slopes remained close to 1; however, the limited number of events and sparse predictor categories affected the stability of bootstrap refitting. Independent validation is therefore needed to confirm the observed performance gains.

Finally, the behavioral data were self-reported and may have been affected by recall or social desirability bias. Several questionnaire items were broad or sparsely populated, and the observational design does not support causal interpretation of the reported associations.

Future research should externally validate the continuous score, deployed categories, and more parsimonious outcome-driven models in larger independent cohorts that include the full risk spectrum and more complete outcome ascertainment. Temporal or geographic validation and assessment of alternative thresholds would help determine whether the optimism-corrected performance gains translate into useful service decisions.

Longitudinal studies should also examine the timing and frequency of venue exposure and incorporate partner- or network-level information. As digital HIVST interventions enter the regulatory landscape, standardized approaches to model validation, updating, privacy, and fairness will also be important [25].

Among MSM who entered the medium- and high-risk online HIVST pathway and had available test results, high-risk venue diversity was associated with observed HIV positivity, and outcome-driven behavioral models achieved higher apparent discrimination than the platform’s continuous rule-based score. After bootstrap optimism correction, the fully adjusted model retained the highest AUC (0.713), and the corrected calibration slopes for both reconstructed models remained close to 1, supporting the potential value of data-driven behavioral weighting for digital HIV risk stratification. The deployed categorical output retained high sensitivity, indicating value for broad risk capture, although its low specificity suggests that the operating threshold could be refined. Taken together, these findings support the feasibility of using routinely collected behavioral and testing outcomes to iteratively evaluate and update digital risk-stratification tools. With further validation, such tools may help online HIVST programs prioritize follow-up testing, tailored counseling, prevention support, and linkage to care, complementing digitally supported HIVST service models reported previously [8,9]. This study therefore provides a practical framework for optimizing rule-based digital assessment systems while preserving accessible self-testing pathways. Because low-risk users and purchasers without usable results were not represented and only 58 positive outcomes were observed, independent validation, broader outcome ascertainment, and prospective assessment of decision thresholds are needed before programmatic implementation. In practice, simply counting the number of distinct high-risk venues a user reports could serve as an easily collectible proxy to prioritize resource allocation for follow-up counseling.