We developed and conducted a pilot study, followed by the main study. The main WEB RDS was initiated on February 28, 2022, and the last participant was introduced on March 28, 2022. The survey closed on April 30, 2022, two weeks after the last questionnaire was completed. We recruited an initial pool of participants (seeds) who were able to invite up to 6 others digitally, who were then able to participate and in turn recruit up to 6 others, ad infinitum. While no incentives were provided during the pilot, during the main study participants received €10 (the conversion rate at the time of the study was €1 to US $1.12) for participation and €5 for each successfully recruited contact [26,30]. Additionally, we halved the survey length from approximately 20 minutes during the pilot to approximately 10 minutes during the main study. Multimedia Appendix 1 presents details of the 2 sampling designs, following the Strengthening the Reporting of Observational Studies in Epidemiology for Respondent-Driven Sampling (STROBE-RDS) report checklist for RDS study design [31].

For the process evaluation, quantitative data such as survey and recruitment data from the pilot and the main WEB RDS study were collected through “National Institute for Public Health and the Environment RDS” software [32]. The software assigns unique identifiers to respondents and uses this to track the recruitment process. The recruitment data consisted of identifier of the “parent” (recruiter of respondent), identifiers of the “children” (recruited by the respondent), number of “children,” and status of the invitations sent (no response, invitation accepted but survey not completed, and survey completed). Respondent network size was measured by asking: “How many Men who have Sex with Men do you know (have contact information of)?” This information was used to compile an RDS data frame. Additional survey data included variables related to socioeconomic status, sexual health, and sexual behavior.

The process evaluation used 4 variables which were all traits suspected to be biased in convenience samples of men who have sex with men, namely age, education, urban residence, and having only anonymous partners. The last variable was included as based on a hypothesis from our formative research; that men who only have anonymous sex might not be reached through WEB RDS. For this trait a proxy was used; only having sex with occasional partners. All variables were categorized into binary variables to aid the graphic analysis, as piloted and described by Gile et al [33].

For the process evaluation, qualitative data were collected as follows: (1) a literature review was performed on WEB RDS recruitment mechanisms related to HICs and the men who have sex with men population; (2) community expert interviews; (3) RDS expert interviews; (4) focus group discussions with community stakeholders were held to better understand the nationwide network structure of Belgian men who have sex with men, the value of incentives optimal for participation and recruitment, and potential barriers and facilitators to recruitment and participation; (5) a pilot study was subsequently performed among the target population; (6) after the pilot, seeds who had not participated or recruited were contacted for a short nonresponse interview (maximum 10 minutes) by phone asking for information on why that was the case. In case individuals recruited by the seed had not participated, seeds were asked why they thought their recruits had not participated; (7) RDS expert consultations and (8) community stakeholder consultations were held to interpret the pilot results and nonresponse interviews, and to develop an incentive structure to be implemented in the study; after the WEB RDS study, (9) further RDS expert consultations and (10) community stakeholder consultations were held to interpret the results. Furthermore, research documents, logbooks, and correspondence between researchers were collected for analysis. For more details on the amount and type of qualitative data collected, see Multimedia Appendix 2.

We used and expanded on the analytical framework developed by Helms et al [19], which itself was adapted from the MRC process evaluation framework [29] to evaluate WEB RDS studies for their systematic review. We made appropriate modifications for evaluating an empirical WEB RDS study (Figure 1) based on the MRC process evaluation framework [29]. We implemented sampling by recruiting men who have sex with men and asking them to invite peers, with the aim that respondents subsequently recruit others into the sample through various mechanisms of impact, such as incentives for participation and recruitment. See Figure 1, adapted from Helms et al [19], for the indicators of implementation, mechanisms of impact, performance, and study setting. A short explanation of each indicator is provided in Multimedia Appendix 3.

To evaluate the indicators listed (Figure 1 and Multimedia Appendix 3), we performed and combined quantitative and qualitative analysis. Survey and recruitment data were analyzed using the RDS package in R (R Foundation for Statistical Computing) and IBM SPSS, adhering to the RDS diagnostics approach developed by Gile et al [33]. Qualitative data were analyzed and codes were generated using a manual thematic inductive approach [34,35]. Themes were categorized into broader descriptive categories, and findings from the quantitative analysis were integrated into the themes at this phase. A coherent line of argument was subsequently generated.

Qualitative data on the indicators of context, namely a HIC with relatively low levels of stigma and good LGBTQI+ rights protection and sexual health services (S1), population network structure (S2), and the sexual health topic (S3), were coded and themes generated. Qualitative data on population network structure (S2) was compared to recruitment homophily (similarity of recruiter and recruited on traits of interest); however, a valid population homophily could not be estimated or evaluated due to inadequate performance. The country (S1) and sexual health topic (S3) indicators were combined with quantitative and qualitative data on incentives (M1), unexpected pathways (M6), and performance (P1-P4, P6).

Implementation was analyzed through social and geographical mapping of the seeds, which consisted of reporting on and analyzing the number of seeds (I4) and their diversity of traits of interest (I3). Qualitative analysis was performed on the recruitment (I1) and selection (I2) process using research notes and a logbook. Synthesis was reached by interpreting the seed mapping results in light of the thematic analysis of our recruitment and selection processes.

Qualitative data related to incentives (M1), random referral (M3), and unexpected pathways (M6) were coded and themes were generated. Furthermore, the effectiveness of incentives (M1) was analyzed by comparing the performance (P1, P3) of the pilot (without material incentives, length 20 minutes) to the main WEB RDS study (with material incentives, length 10 minutes). These findings were then integrated into the thematic analysis. There were insufficient data for qualitative or quantitative analysis of recruitment options (M2), coupons (M4), and reminders (M5).

First, we evaluated whether the total number of participants recruited (P1) was sufficient to compensate for the RDS design effect of 3 with an SE of 0.03, as most recently recommended for RDS [36,37]. We calculated the minimum sample size (Multimedia Appendix 4) using the formulae outlined by Salganik [37], and this resulted in a target sample size of 625 participants. The calculation can be found in Multimedia Appendix 4. Second, we examined participants who were recruited successfully (P2) through participation and recruitment rates. Third, we compared the number of waves (P3) to standards for sufficient waves in the field. Fourth, the dynamics of the RDS estimates were explored using a graphical approach to determine if the estimates had stabilized without bottlenecks (P4) [33]. This was carried out by a visual examination of the convergence plots (estimates over time for the total sample), bottleneck plots (estimates over time per chain), and all-points plots (inclusion of traits over time), as suggested by Gile et al [33]. The estimator used was RDS-II, also called the Volz-Heckathorn estimator [13].

Finally, to evaluate the inclusion of subpopulations (P6) of interest, we compared sociodemographics of the WEB RDS sample to sociodemographics of a cross-sectional convenience sample collected on the web from the European Men Who Have Sex With Men Internet Survey (EMIS) [38]. Data on the duration of the data collection indicator (P5) were not informative and hence were not analyzed in-depth.

Thematic analysis of the indicators of mechanisms of impact resulted in rich data on incentives (M1), random referral (M2), and unexpected pathways (M6). Both RDS and community experts advised us to provide material incentives (M1). Community experts framed the need for incentives to show appreciation for the target population, create goodwill, show that their participation is not taken for granted, and compensate people for their valuable time. Both community and RDS experts advised against a lottery ticket or a very small sum because they expected that would be perceived as insulting and underappreciative. The monetary incentive structure and amount (double incentive system, value ranging €10-€30 as an electronic gift card) were based on literature [26,30] and positive feedback from the community representatives. We investigated the impact of monetary incentives by comparing results from the pilot (without monetary incentives; 20 minutes) to the main WED RDS study (with monetary incentives; 10 minutes). The main study clearly outperformed the pilot: the pilot started with 24 seeds and ended spontaneously with 25 completed surveys after 3 waves. The main study began with 38 seeds and ended spontaneously with 151 completed surveys after 7 waves. We suspect the higher performance is due to the incentive provided. Although the survey length of the main study was half that of the pilot (10 minutes compared to 20 minutes), qualitative research did not indicate that this made an impact. Survey length was not mentioned as a barrier in the nonresponse interviews and there was little dropout while filling out the survey during the pilot. It is possible that the survey length had an invisible effect on the willingness to recruit; however, this could not be investigated further with the data available.

During consultations and project meetings, RDS experts advised us to forego randomization of recruitment (M2), despite it being an RDS assumption. They stated that randomization has proven too much of an obstacle for respondents and that random selection of contacts from a list is not very realistic, because people are reluctant to send invites to random contacts. Consequently, too many branches die off prematurely (eg, there is an insufficient number of waves [P3]). Community experts confirmed that randomly selecting a recruit from a list of (pseudonym) names was an unrealistic expectation. Not requiring random recruitment (M2) likely led to recruitment bias, and providing incentives (M1) may have exacerbated this bias. RDS experts stated that people sent invitations to those they believe will participate, to increase the chance of getting paid. They hypothesized that well-known activistic men who have sex with men with a large network are perceived as more likely to participate, and therefore, would be oversampled. We had insufficient quantitative data to confirm or deny this hypothesis.

Two unexpected pathways (M6) were identified: (1) survey fatigue, in the sense of being overresearched as a population in general (S2) and on the topic of sexual health (S3) in particular, and (2) digital etiquette discouraging sharing survey links. Community experts stated that men who have sex with men contacted by their organizations experienced general survey fatigue. The issue of survey fatigue and the overwhelming number of survey invitations encountered by the population were confirmed during nonresponse interviews after the pilot.

In addition to a general unwillingness to participate in surveys, topic-specific (S3) unwillingness was also reported. Community experts underlined that survey fatigue was especially high in sexual health surveys. They highlighted that gay men resent the unwarranted focus on their sexual practices, rather than on other aspects of their lives. During non-response phone interviews, respondents confirmed no motivation for the topic as one of the reasons for nonresponse (P2) among their contacts. The survey topic (S3) was also considered a barrier to recruiting others (P2); during non-response phone interviews after the pilot, men who have sex with men stated that they had not invited others due to the perceived complications and sensitive nature of sharing a survey on sexual health.

RDS experts also anticipated a cultural resistance to sharing survey links; digital etiquette discouraging spamming contacts in general and with survey requests in particular. Community organization representatives also expected digital etiquette to be a barrier to recruitment (P2) among Belgian men who have sex with men. They advised providing incentives (M1), as this could change the invitation from “spam” into “opportunity” and make people more comfortable inviting others to a web-based survey.

Our results on short recruitment chains align with those of Strömdahl et al [26], which suggests that WEB RDS with the goal of producing population estimates may not be viable among men who have sex with men in HICs, as long recruitment chains are necessary for valid inference [26]. Similar to Strömdahl et al [26], our study faced significant barriers to long recruitment chains, mainly nonresponse, likely linked to extreme survey fatigue among our target population. Further research into the extent and mechanisms behind survey fatigue among men who have sex with men is needed, not only for studies using WEB RDS but for other survey methods as well. Survey fatigue may be in line with a wider societal trend of web-based survey oversaturation and declining response rates [41-43]. In light of this, future studies could increase the number of coupons (more than 6) to compensate for the low survey participation rate and increase the sample size and length of recruitment chains. Survey fatigue may also be due to overresearch, where a particular population attracts disproportionate attention [44]. The latter could be an artifact of men who have sex with men being designated an at-risk population. The mechanism could be similar to spatial bias, where certain areas are more prone to being researched because of particular features of their populations, which in turn creates resistance to research within those populations [45]. If men who have sex with men–specific research fatigue exists, coordinating research among men who have sex with men within and among universities, governments, and community organizations could address this and would depend on political and social alignment between these organizations regarding research goals and aims.

Furthermore, topic-specific research fatigue among men who have sex with men is an additional concern for future studies. In our study, community organization representatives reported that their members were unhappy with the sheer number of studies conducted on their sexual behavior, versus other important aspects of their lives, and were thus unlikely to participate in studies on sexual health. The perception that researchers are driven by different agendas for their research populations can produce frustration with high levels of research attention and can lead to high levels of nonparticipation [46,47]. A study on nonresponse bias in surveys on HIV and sexuality found that topic salience plays a large role, with 20% declining to participate due to a lack of interest in the topic [48].

No motivation for the topic could also be a crucial barrier to respondent-driven recruitment; as men who have sex with men in a study by Diexer et al [30] indicated that motivation for the topic was the most important consideration for inviting contacts to a WEB RDS [30]. This motivation was considered far more important than a sufficient reward for inviting others. A systematic review of response rates to web-based surveys among men who have sex with men on various topics may shed further light on the research needs of men who have sex with men and their motivation for specific topics, without further overburdening the population. A preparatory, qualitative study into the perception among men who have sex with men about their research needs versus research encountered, their research participation history, and their preferred recruitment and data-sharing methods could also pave the way for future successful sampling.

The value of the incentive to be provided in a WEB RDS is a delicate balance. We found that €10 for participation and €5 per successful recruitment, up to a total of €30, was insufficient to overcome barriers among Belgian men who have sex with men. On the other hand, Wejnert et al [18] found that total compensation of US $55 was too high among American students, as it was perceived as a potential scam [18]. A similar study by Bauermeister et al [20] among American youths found that US $70 was successful for recruitment compared to US $30, which was not [20]. A study in Thailand showed that a WEB RDS could be performed there without incentives because contributing to research is culturally considered to not need a reward [49]. The need for a material incentive and a reasonable amount are likely country-, culture-, and population-specific. The fact that our main WEB RDS (with incentives) performed better than our pilot (without incentives) indicates that some form of material incentive is crucial for the sampling mechanism to work among men who have sex with men in Belgium. It remains to be seen, however, if increasing the amount would lead to more successful recruitment, as there are limits to what a conditional incentive can achieve. Marginal gains in participation decrease as the incentive gets higher—a participation plateau will be reached even when further increasing the incentive [50]. Our formative research indicates that a “reasonable” incentive functions as a sign of respect from researchers towards men who have sex with men; it shows that their participation is not taken for granted, which may be crucial when sampling populations with high levels of survey fatigue.

During consultations and project meetings with WEB RDS researchers and community organization representatives, we noted the importance ascribed to “Netiquette”—digital social norms that govern digital communication undertaken with contacts [51]. Netiquette denoted survey requests as a type of spam. Research into “Netiquette” and the acceptability of receiving a survey request from a contact, may further clarify the effectiveness of WEB RDS for studying men who have sex with men and other populations Leverage saliency theory suggests that the wording of the request and which aspects of the study are emphasized in the request are crucial [52], and this effect could be even stronger among populations encountering a large number of surveys invitations and experiencing survey invitations as spam. An accessible way to study the impact of invitation phrasing by respondent recruiters would be to ask respondents to share their invitation texts and compare those to the (non-) response.

One major benefit of WEB RDS compared to offline RDS is access to a digitally connected population spread over a larger geographical area, such as a country [18]. This can, however, present challenges for population inference, as an important requirement is that the entire population under study is part of the same social network [33]. We found that community experts did not think men who have sex with men across the nation to be one network, but rather that networks are confined to a province or city. Recruitment homophily on the living environment (city or rural) was found to be modest. The qualitative findings are in line with those from WEB RDS studies in Sweden and Vietnam on geographical clustering [17,26]. A high degree of clustering implies that longer recruitment chains are needed for estimates to stabilize than previously thought [33]; this may be a bottleneck for nationwide WEB RDS of men who have sex with men in HICs, as reaching long chains has proven problematic [26,53,54]. Future studies should include a measure of location to monitor recruitment and determine if population inference over the desired geographical area is viable.

The fact that the population at increased risk of HIV and sexually transmitted infections (such as closeted men who have sex with men and men who have sex with men who have sex with anonymous partners) may not be reached through WEB RDS is especially relevant in the field of sexual health [55-58]. Community organization representatives did not expect these populations to be reached. Our WED RDS diagnostics did not show recruitment bias or differences in network size among men who have sex with men with occasional partners, and these men who have sex with men were represented in the final sample. However, as a proxy for men who have sex with men with only anonymous partners, it has its limitations. Offline RDS studies have been able to include men who have sex with men who were not out or open about having sex with men [54] and one offline RDS was able to include more men who have sex with men who did not identify as gay than a simultaneous time-location-(venue)-based sampling [59]. At least a portion of men who have sex with men who have anonymous sex contact with their sex partners through the internet, and this could facilitate WEB RDS [58]. Future WEB RDS among men who have sex with men aiming to include these populations may want to include variables on the level of outness, anonymity of sex partners, sharing contact information with other men who have sex with men, and how often they meet other men who have sex with men in person. An offline RDS or hybrid form could be explored to include those men who have sex with men who do not share contact information with other men who have sex with men.

The unique respondent-driven recruitment mechanism of WEB RDS likely led to some differences in sample composition compared to the EMIS-convenience sample. Our results show that men who have sex with men recruit those similar in age and urban environment, indicating that the composition of a WEB RDS sample (one that does not reach stabilized estimates) can be controlled through characteristics of the first respondents in the chain. Further studies are needed to determine how WEB RDS recruitment behavior impacts sample composition. Combining web-based convenience sampling with selective WEB RDS should be investigated further; when bias in web-based convenience sampling is expected, participants with underrepresented traits could be asked to recruit others through WEB RDS to mitigate this bias.