All data were collected as part of a randomized controlled trial (RCT) of Outsmart HPV. The methods of the RCT have been described previously [7] and are briefly summarized below. We recruited a convenience sample of young gay, bisexual, and other men who have sex with men via social media sites, dating apps, and other web-based avenues. Advertisements on these digital platforms included a combination of images (eg, men in the targeted age range) and brief text about the study (eg, information about HPV, the study being web-based). Although we standardized advertisement content where possible, some content did differ across platforms due to varying advertisement requirements and options (eg, amount of text allowed, advertisement placement). For example, some of our advertisements through dating apps featured content that was sent directly to users’ in-app inboxes over a 24-hour period, which was an approach that was not available across all platforms.

Interested individuals were linked via advertisements to a mobile-friendly project website to complete an eligibility screener. Eligibility criteria included self-identifying as (1) cisgender male; (2) aged 18-25 years; (3) either gay, bisexual, or queer; ever having oral or anal sex with another male; or being sexually attracted to other males; (4) living in the United States; (5) not having received any doses of HPV vaccine; and (5) not having previously participated in Outsmart HPV. Eligible individuals provided informed consent and created a project website account.

Participants next completed a baseline survey (“T1 survey”) and were then immediately randomized using a 1:1:1 allocation scheme to 1 of 3 study groups. The 3 study groups included 2 groups that received intervention content and 1 control group [7]. Participants then viewed intervention content about HPV vaccination on the project website, with participants in both intervention groups viewing Outsmart HPV content about HPV vaccination and participants in the control group viewing standard information about HPV vaccination. After viewing the intervention content, participants completed a second survey (“T2 survey”). Additional follow-up surveys occurred 3 and 9 months later (“T3 survey” and “T4 survey,” respectively). All study surveys were completed on the project website. A total of 1227 participants were randomized from October 2019 to June 2021.

The main independent variable for this report was the type of web-based recruitment source, with each participant categorized as being recruited from either: (1) social media (eg, Facebook, Instagram, Snapchat); (2) a dating app (eg, Grindr, Scruff); or (3) some other recruitment source (eg, existing research panel, university-based organization).

The T1 survey collected information on a range of demographic and health-related characteristics. This included participants’ disclosure of their sexual orientation (3 items; α=.77; possible range 1-5) [34], concealment of their sexual orientation (3 items; α=.75; possible range 1-5) [34], connectedness to the LGBTQ (lesbian, gay, bisexual, transgender, queer) community (2 items; α=.72; possible range 1-4), and electronic health literacy (4 items; α=.83; possible range 1-5) [35]. Items were coded so that higher values indicate greater levels of a given construct. We examined the type of device that participants used during study enrollment, with each participant categorized as using either a smartphone, tablet device, or a personal computer (eg, desktop, laptop).

To examine study retention, we determined whether participants completed the T2, T3, and T4 surveys. For each survey separately, participants were categorized as having either completed the survey or not completed the survey. Among participants in the 2 intervention groups, we also examined additional metrics of study engagement by calculating the total amount of time (in minutes) spent viewing Outsmart HPV content about HPV vaccination and the total number of logins to the project website. Both of these metrics were winsorized so that the top 10% of values were set to equal the value corresponding to the 90th percentile.

We first calculated descriptive statistics for all variables. We examined differences by recruitment source in participants’ demographic and health-related characteristics and device type. To make these comparisons, we used the Pearson chi-square test for categorical dependent variables and 1-way ANOVA (analysis of variance) for continuous dependent variables. We made post hoc pairwise comparisons between the 3 web-based recruitment source groups and used the Bonferroni adjustment to account for multiple comparisons.

We then used regression models to examine differences by recruitment source in study retention and other study engagement metrics. Logistic regression models produced odds ratios (ORs) and 95% CIs for completion of the T2, T3, and T4 surveys. Linear regression models produced standardized β coefficients for the other study engagement metrics. The recruitment source of dating apps served as the referent group in regression models. These statistical tests were 2-tailed with a critical α value of .05. Data were analyzed using R (version 4.2.1; R Foundation for Statistical Computing), and Stata (version 15.0; StataCorp).

The institutional review board at The Ohio State University approved this study (IRB 2019C0028), and the RCT is registered at ClinicalTrials.gov (NCT04032106). We based the reporting of results for this manuscript on the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines [36] since most of the included data were collected at baseline. Informed consent was obtained from all participants, and all data were deidentified. Participants could earn up to US $95 in gift cards for completing study surveys.

Overall, 52.8% (n=648) of participants indicated a minoritized racial or ethnic identity, 64.4% (n=790) were between 22 and 25 years of age, 69% (n=847) had at least some college education, and 66.4% (n=815) self-identified as gay. Most participants (n=974, 79.4%) had some form of health insurance, but more than half (n=666, 54.3%) had not had a preventive health visit in the prior year.

A total of 403 (32.8%) participants were recruited via social media, 683 (55.7%) participants via dating apps, and 141 (11.5%) participants via other recruitment sources. Most demographic and health-related characteristics differed by recruitment source (Table 1). Generally, a higher proportion of participants recruited via dating apps reported a minoritized racial or ethnic identity, were single and having sex or casually dating, had less education, reported having sex with a male partner in the past, and had public or no health insurance. Conversely, a higher proportion of participants recruited via social media indicated not being currently employed full-time or part-time, self-identified as gay, and reported being HIV-negative. Participants recruited via social media also reported a higher level of connectedness to the LGBTQ community and a higher level of disclosure of their sexual orientation. Table 1 shows the results of all pairwise comparisons.

The type of device used by participants during study enrollment also differed by recruitment source (P<.001; Table 1). Smartphone use was higher among participants recruited via dating apps (n=660, 96.6%) compared to those recruited via social media (n=318, 78.9%) or other web-based sources (n=85, 60.3%). Conversely, personal computer use was higher among participants recruited via social media (n=81, 20.1%) or other web-based sources (n=55, 39%) compared to those recruited via dating apps (n=17, 2.5%).

Participant retention overall was 90% for the T2 survey (n=1001), 70% for the T3 survey (n=858), and 69% for the T4 survey (n=842). Compared to participants recruited via dating apps, retention was higher among participants recruited via social media for the T2 survey (94% vs 87%; OR 2.09, 95% CI 1.32-3.30), the T3 survey (75% vs 66%; OR 1.52, 95% CI 1.15-2.00), and the T4 survey (75% vs 65%; OR 1.66, 95% CI 1.26-2.19; Figure 1). There were no statistically significant differences in retention for participants recruited via other web-based sources.

Among participants in the 2 intervention groups (n=815), the average amount of time spent viewing intervention content about HPV vaccination was 2.83 minutes. Participants recruited via social media spent a longer amount of time viewing this content compared to those recruited via dating apps (3.14 minutes vs 2.67 minutes; 95% CI 0.03-0.33; P=.02; Table 2). The average number of project website logins was 4.94, and there were no differences in the number of logins by recruitment source (P=.83 for social media, P=.54 for other web-based sources; Table 2).

With digital methodologies becoming increasingly popular for recruiting young gay, bisexual, and other men who have sex with men for research studies, [17-21] it is important to examine if different web-based recruitment sources yield varying types of participants. The Outsmart HPV study provided an opportunity to address this research topic by recruiting a national sample of young gay, bisexual, and other men who have sex with men through multiple web-based sources, and there are 2 main findings. First, most of the demographic and health-related characteristics examined differed by recruitment sources. Participants recruited via social media tended to be less diverse (eg, race or ethnicity, sexual self-identity), of higher socioeconomic status (eg, education level, health insurance status), and have higher levels of outness (eg, disclosure of sexual orientation, connectedness to the LGBTQ community) compared to participants recruited via dating apps. This pattern differs from a past study that found participants recruited from a social media site (Facebook) were highly similar to those recruited from a dating app (Grindr) in terms of race or ethnicity [18]. Interestingly, the characteristics of participants recruited via social media in our study better align with the characteristics of other national samples of young gay, bisexual, and other men who have sex with men and the larger gay, bisexual, and other men who have sex with men community in the United States than participants recruited via dating apps in our study [37-40]. A potential explanation for this pattern is that users of dating apps tend to be more demographically diverse compared to the general population [41]. Interestingly, some characteristics (eg, socioeconomic status) of participants that were recruited via social media for our study were more comparable to gay, bisexual, and other men who have sex with men recruited through in-person venues in past research than via dating apps in our study [42].

Second, study engagement also differed by recruitment source. Participants who were recruited via dating apps tended to interact less with study content and were less likely to complete follow-up surveys compared to participants recruited via social media. This may be attributable to users of dating apps having an emphasis on immediate gratification [43] and impulsivity [44] as a justification for their app usage (and study enrollment), which in turn may affect their awareness of and interest in engaging with study content and completing study-related activities (eg, follow-up surveys). Related, given that dating app users are more impulsive than nonusers, [44] young gay, bisexual, and other men who have sex with men who use dating apps might experience an initial interest in joining a research study and earning an initial study incentive. However, interest in completing study-related activities at later timepoints may not be as strong as their general preference for some level of anonymity [45] might dissuade them from wanting to continue discussing their health behaviors (eg, sexual history, vaccination status).

We think these 2 main findings can help researchers better plan and customize future digital recruitment efforts of young gay, bisexual, and other men who have sex with men by highlighting key issues and considerations. For example, researchers will now be able to better anticipate how the selection of web-based recruitment sources may affect the representativeness and diversity (and the potential balance between the two) of their study sample. Recruiting through social media may lead to a sample of participants that better reflect the larger young gay, bisexual, and other men who have sex with men population in the United States, but dating apps may be more effective in reaching more diverse young gay, bisexual, and other men who have sex with men, including those from communities that are often underrepresented in research studies [46]. Recruitment goals differ across studies in terms of representativeness and diversity, and our data on differences in demographic and health-related characteristics can help inform future efforts. Furthermore, studies involving interventions and/or longitudinal data collection should consider how various web-based recruitment sources can impact study retention and participants’ engagement with study content. Some studies may want to maximize participant retention and engagement (ie, suggesting that recruitment primarily through social media may be appropriate), whereas other studies may be willing to have slightly lower study engagement metrics in exchange for being able to include additional recruitment sources, such as dating apps. Future studies that do include recruitment from dating apps may also want to consider prioritizing brief study-related activities and other strategies to promote engagement.

One final implication involves the types of devices that were used by participants in our study. Overall, a majority of participants used a smartphone to enroll in the study, including nearly all participants who were recruited via dating apps. This pattern is similar to national data showing that young adults use smartphones much more frequently than other device types [47,48]. Thus, it is important that future digital interventions and other research studies for young adults, including young gay, bisexual, and other men who have sex with men, ensure that study content and data collection instruments are optimized for use on smartphones. For example, to increase usability, it has been suggested that the interface of smartphone-based interventions have a minimal number of screens, limited manual data entry, limited pop-ups and notifications, and personal identification number-based entry (as compared to full passwords). The content of such interventions should be succinct with easy-to-read graphics [49]. Data collection instruments (eg, surveys) should have limited open-ended questions and different “pages” in the survey, rather than having participants scroll through one continuous page. Lastly, and perhaps most generally, mobile surveys should be pilot-tested prior to the start of data collection activities [50]. As smartphone usage continues to be a common approach for participants enrolling in and interacting with digital interventions, such recommendations can help optimize intervention delivery and data collection.

Although we were able to compare different groupings of web-based recruitment sources, we were not able to examine potential differences between individual platforms (eg, Facebook vs Twitter vs Grindr) due to modest sample sizes for some of the platforms. These sample sizes also did not allow us to examine how differences between recruitment sources may have changed over time during the recruitment period for our study. Examining potential differences between individual platforms and potential temporal changes are important areas for future research. Our recruitment focused on the social media and dating apps that were among the most popular among young adults in the United States around the time of the start of our study [51,52]. It is likely that the use and popularity of various platforms will evolve over time, and future efforts should examine new and emerging platforms. We were not able to examine advanced metrics related to recruitment efficiency and cost due to a lack of available data. However, past studies have reported such information for various web-based platforms [17,27]. Study recruitment occurred both prior to and during the COVID-19 pandemic, and it is not known how the pandemic may have affected the interest of young gay, bisexual, and other men who have sex with men to enroll in a study focusing on vaccination (which would subsequently affect the characteristics of study participants). Fraudulent accounts with inaccurate self-reported data occur in digital studies, but we used several recommended strategies for detecting such accounts and minimizing this risk [53,54].

We were able to recruit a large national sample of young gay, bisexual, and other men who have sex with men for an HPV vaccination intervention via multiple web-based methodologies. Participants differed on a range of demographic and health-related characteristics based on whether they were recruited from social media, a dating app, or some other web-based recruitment source. Study retention and engagement with study content also differed by web-based recruitment source. These findings and patterns highlight key issues and potential tradeoffs that can help researchers better plan and customize future web-based recruitment efforts of young gay, bisexual, and other men who have sex with men.