The data set was derived from a 3-arm randomized controlled trial of secondary distribution of HIVST kits in Zhuhai, China [19]. This trial was registered with the Chinese Clinical Trial Registry (ChiCTR1900025433). All participants gave digital written informed consent by providing electronic signatures before the taking the web-based baseline survey. Between October 21, 2019, and September 14, 2020, a 3-arm randomized controlled, single-blinded trial was conducted on the web among 309 individuals (defined as “index participants”) who were assigned male at birth, aged 18 years or older, ever had male-to-male sex, willing to order HIVST kits on the internet, and consented to take surveys on the web. In this trial, 309 MSM were randomly assigned to the control group (standard secondary distribution [SD] arm; SD group), the intervention I group (SD with monetary incentives [SD-M] arm; SD-M group), or the intervention II group (SD-M and peer referral [SD-M-PR] arm; SD-M-PR group). Monetary incentives implies that the index participants in the SD-M and SD-M-PR groups could receive a fixed incentive of US $3 on the web for a verified test result uploaded to the digital platform by each unique alter. Monetary incentives and peer referral implies that the index participants in the SD-M-PR group could additionally have a personalized peer referral link for alters to order kits on the web as an intervention strategy. Of 309 indexes, 60 were key distributors who passed the kits to at least 2 alters. Additionally, there were 73 key promoters, leading to 103 alters who were first-time testers; and 23 key detectors, leading to 25 alters who were undiagnosed people living with HIV, as defined above. The trial profile infographic with more details can be found in Zhou et al [20].

We formulated a strategy to identify key influencers as a binary classification problem, and 4 ML models were constructed, including logistic regression, support vector machine, decision tree, and random forest [21,22]. Each model has its pros and cons (Multimedia Appendix 1) in performing the classification task; hence, we comprehensively combined the predictions of all 4 models to mitigate overfitting and model biases by adopting an ensemble learning approach [23], which could synthesize the strengths from each ML model. Specifically, we used the voting classifier in soft mode as the ensemble method, considering the probabilities yielded by each ML model, and these probabilities would be weighted and averaged; consequently, the winning class would be the one with the highest weighted and averaged probability.

To evaluate ML models for such classification tasks, we used 4 metrics for performance evaluation: accuracy, precision, recall, and F₁-score (Multimedia Appendix 2). Accuracy is defined as a ratio of correctly predicted observations to the total number of observations. The F₁-score takes both the precision and the recall into consideration and is defined as the harmonic mean of precision and recall:

We used 5-fold cross-validation [24] to ensure the robustness of the models and compared the average values of each metric. Specifically, we randomly sampled 80% of the data for training and 20% for testing. Experiments for each metric were repeated 5 times as every time 1 fold (ie, 20% of the data) would constitute the testing set and the remaining 4 folds (ie, 80% of the data) would be trained for model construction and parameter learning. The final average values of each metric are the average performance of the 5 folds’ testing set.

First, we incorporated original predictors (ie, input variables from the survey) into our classification models using the aforementioned 4 ML models. Then, we obtained a predictor ranking list of each ML model ordered by the importance of every variable (Multimedia Appendix 3). The 4 ML models voted for the final selected predictors, which ranked at the top in all 4 importance ranking lists.

Specifically, the logistic regression model provided us with the coefficient of each predictor, while the other 3 models (support vector machine, decision tree, and random forest) provided the importance level. Specifically, in logistic regression, we ranked the importance of each characteristic in accordance with the absolute value of the standardized coefficient. For the support vector machine model, we adopted the Recursive Feature Elimination algorithm to generate a weighted vector when training, and then in each iteration, we eliminated a least important feature through the above-weighted vector. For decision tree and random forest models, the Gini index in the Classification and Regression Tree algorithm determined the importance of every variable.

After determining the top predictors by importance ranking, we ran the same 4 ML models based on these selected variables to check and obtain the classification performance. Then, we used ensemble learning to combine the findings from all ML models. The whole process, including ML modeling, predictor selection, and modeling based on selected important variables (ensemble learning), represents a novel intelligent identification system (illustrated in Figure 1), which can be adopted in the implementation program in the secondary distribution of HIVST kits in the future to identify key influencers among MSM.

We ran all ML experiments in Python (version 3.7; Python Software Foundation), and the code is available upon request from the corresponding author.

For comparison with our findings using ML models, we used 2 self-reported scales as the human identification approach. According to existing literature, self-evaluated leadership scales are commonly used to identify key influencers [11,12,14]. These self-reported leadership scales asked indexes to evaluate the likelihood of 6 social influence–related scenarios on a scale of 0 to 4, and the total points ranged from 0 to 24. Based on the previous literature, indexes (around 20% out of a total of 309 indexes) who distributed at least 2 kits were defined as key distributors on the ground truth. Hence, we set a cutoff of the top 60 indexes (which was also around 20%) by rank order in our 6-question self-reported scales (Figure 1) as the human-identified key influencers. However, 49 indexes received at least 11 points in the self-reported scales, while 81 indexes received at least 10 points. In other words, since the 49th to the 80th indexes received the same points in these self-reported scales, we were unable to determine exactly which of these indexes ranked in the top 60. As a result, we regarded these 2 scale cutoffs as human identification baselines together, recorded as cutoffs A and B, respectively.

Finally, we further conducted a simulation model to mimic the secondary distribution process on the MSM’s social network and to compare the intervention efficiency of identification of the key influencers by the ML models and by conventional human identification approaches. Here, intervention efficiency is defined as the number of individuals who have self-tested at the end of the simulation. Specifically, simulation technologies [25,26] on HIV-related networks [27,28] can also model distribution network characteristics.

We simulated the secondary distribution process on each test set of the 5-fold cross-validation. Given indexes in each test set, we constructed a network containing both indexes and alters who received self-testing kits from these indexes. There would be an edge between an index and an alter if the alter received a kit with the corresponding confirmation code of the index. Self-testing kits would be distributed through edges on the network. Specifically, we summarized and interpreted the secondary distribution process observed from our empirical studies [20,29] into a simplified diffusion model. We used a Poisson distribution to mimic the distribution behavior. Only 1 parameter is needed in the Poisson distribution: the mean number of HIVST kits an individual wants to distribute at each time step. We set it as the number of received HIVST kits. The number of HIVST kits allocated to indexes is set at 4, as the number of HIVST kits an index can order in our empirical experiments is generally no more than 5 [20]. The code for simulations is available upon request from the corresponding author.

Ethics approval of this trial was obtained through the Zhuhai Center for Disease Control and Prevention (ZhuhaiCDC-201901) [19]. All participants provided written informed consent.

We compared the performance of ML classification and ensemble learning with that of the human identification approach based on leadership self-evaluated scales in terms of the 2 nearest cutoffs. In our survey data, 60 (19.4%; ie, around 20%) indexes who distributed at least 2 kits were key distributors on the ground truth. In addition, these key distributors reached more than 70% of alters in total. Additionally, there were 73 key promoters who helped us promote first-time testing to 103 alters and 23 key detectors who helped us detect 25 positive alters.

Table 1 shows that ML classification significantly outperformed human identification cutoffs irrespective of the type (ie, key distributors, key promoters, and key detectors) adopted to define the key influencers (technical details provided in Multimedia Appendix 1). The model using ensemble learning also outperformed the human identification approach and nearly achieved the highest value among all models in terms of the performance metrics. Specifically, for 3 classification training rules (ie, 3 types of key influencers), the classification performance of ensemble learning obtained an accuracy of 90%, 93%, and 82% for key distributors, key promoters, and key detectors, respectively, all exceeding human identification (the cutoffs of the self-reported scales). Therefore, the ensemble learning approach combining 4 ML models could better capture key influencers, compared with the other approaches studied, with an 11.0% higher accuracy on average than human identification approaches.

Table 2 compares the number of distributed kits from key influencers identified using the ensemble learning model and the human identification approach. Both the ensemble learning model and the human identification approach classified 49 key influencers each, but those identified using the ensemble learning approach distributed 146 kits, equating to 54% (146/269) of alters. In contrast, the 49 key influencers identified using the human identification approach only distributed 97 kits. In addition, the same 49 key influencers identified by the ensemble learning model identified 3 more people living with HIV and 14 more first-time testers than those identified using the human identification approach. In summary, our new approach could identify the distribution of 18.2% (95% CI 9.9%-26.5%) more kits, 13.6% (95% CI 1.9%-25.3%) more first-time testing alters, and 12.0% (95% CI –14.7% to 38.7%) more undiagnosed people living with HIV.

We simulated the secondary distribution process on each test set of 5-fold cross-validation. The simulation results (Table 3) show that our ensemble ML approach could always obtain a higher intervention efficiency in each fold than the conventional human identification approach. Specifically, the average intervention efficiency of the ensemble ML model increased by 17.7% (95% CI –3.5% to 38.8%) compared to that of the self-reported scales cutoff method, which indicates a higher intervention efficiency of our novel method to identify key influencers.

As shown in Table 3, we observed a higher distribution efficiency for ML models than for conventional human identification approaches. More technical details of this simulation are shown in Multimedia Appendix 1.

Identifying key influencers for secondary distribution of HIVST kits among Chinese MSM is important. Identification of key influencers who may be more active in the secondary distribution of HIVST kits can potentially expand testing coverage, reach more naïve testers, and help identify undiagnosed people living with HIV. We found that using an ML approach, specifically ensemble learning, was superior to human identification of key influencers (Figure 2).

We found that all 4 ML models outperformed human identification. Our results are consistent with those of other reports that ML models have performed better in other HIV-related identification tasks such as identifying populations at a high risk of HIV [17], identifying HIV-related social media data [30], and identifying people eligible for PrEP [18,31]. This adds to the evidence demonstrating that ML can efficiently identify key influencers within networks compared to other methods.

In addition, we found that key influencers identified by our ensemble learning approach could distribute 18.2% more kits, identify 13.6% more first-time testing alters, and detect 12.0% more undiagnosed people living with HIV than the conventional human identification approach. Regarding why our ensemble learning approach outperformed the human identification approach [14], we believe this may be because the ML algorithms included variables related to 2 key drivers of identification: men's HIV testing and kit application. Self-reported leadership scales are not specifically designed to consider such important predictors. Our data suggest that ML could enhance the accuracy of social evaluation scales for identifying key influencers. Using an ML approach could significantly improve the public health impact of secondary distribution.

Our method offers a potential means to prioritize indexes identified as key influencers in the secondary distribution of HIVST kits using ML. Our novel ensemble ML approach for identifying key influencers in the secondary distribution of HIVST kits can accurately and rapidly classify which indexes are crucial for distributing more kits, promoting testing among novice testers, or detecting more undiagnosed people living with HIV. This is especially significant for low- and middle-income countries where resources for HIV testing services may be limited.

Our study also has several limitations. First, our study had a retrospective modeling design, and we are currently conducting a prospective trial to compare ML and the conventional method [29]. Second, due to the survey content, we only compared our ML approach with 1 type of human identification, namely, self-reported leadership scale cutoffs. Future studies should explore comparisons of the ML approach with other methods of human identification. Third, the sample size (ie, 309 indexes) was relatively small for ML modeling, which could be considered another significant limitation of this study.

In conclusion, we found that ML using ensemble learning achieved the highest accuracy in identifying key influencers who are more effective at secondary distribution of HIVST kits in China (Figure 2). Therefore, with regard to our future research plans, we are currently implementing this approach in a program involving the distribution of 2000 HIVST kits through a quasi-experimental trial comparing ML identification to scales-based human identification [29]. Our ensemble learning approach can also be generalized to identify key influencers for other HIV prevention and treatment programs.