Reports eligible for inclusion in the overall review reported on eHealth interventions that were delivered by mobile phone apps, the internet, or other electronic communication technology; offered ongoing support to populations consisting entirely or principally of gay, bisexual, and other men (including cisgender and transgender men) who have sex with men; and aimed to prevent HIV/sexually transmitted infections, sexual risk behavior, alcohol, tobacco or drug use, anxiety, or depression. We excluded interventions that offered one-off (rather than ongoing) support or that involved human providers (eg, in a chat room). Reports eligible for the process evaluation synthesis reported on characteristics of interventions, providers, participants, or context affecting delivery or receipt of eligible interventions. We included published and grey literature and set no restrictions by location or language.

Terms used in our search strategy covered 2 concepts joined by the Boolean operator “and”: MSM and eHealth. We searched 19 databases containing health and social science literature (from October 2018 to November 2018 and updated on April 2020). Our complete search strategy for the original OvidSP MEDLINE database is included in Multimedia Appendix 1 and the search strategies for all databases are available at the London School of Hygiene & Tropical Medicine’s Data Repository [35]. We conducted additional searches by searching 3 clinical trials registers, the OpenGrey database, and Google (limited to first 100 results), and by completing reference checks of included reports and requests from experts.

Citations were uploaded to EndNote (Clarivate Analytics), deduplicated, and then uploaded to EPPI-Reviewer (version 4.0, EPPI-Centre) for screening. CB and JF independently screened titles and abstracts in batches of the same 50 references, resolving disagreements by discussion. After reaching an agreement rate of at least 95%, they single-screened all remaining citations. Screening of full texts followed a comparable process.

For process evaluations, CB and RM used an adapted version of an existing tool [36] to independently extract data reporting empirically on how processes of delivery or receipt varied with characteristics of interventions, providers, participants, or contexts. They also extracted data on basic study details, methods, and intervention descriptions. They then independently assessed the quality of process evaluation reports using standard Critical Appraisal Skills Program and EPPI-Centre tools [37]. These addressed the rigor of sampling, data collection and data analysis; the extent to which study findings were grounded in the data; whether the study privileged the perspectives of participants (eg, by including open-ended questions or otherwise allowing space for participants to set out their own views); and the breadth and depth of findings (ie, the extent to which the study explored a broad range of process issues or provided in-depth insights into participant perspectives). Drawing on these criteria, each reviewer then assigned weights (high, medium, or low) to rate the reliability or trustworthiness of the findings, and the usefulness of the findings for shedding light on the research question (ie, the extent to which they shed light on how processes of intervention delivery and receipt varied with characteristics of interventions, providers, participants, or contexts). Reliability reflected the trustworthiness of the overall findings (ie, the extent to which the methods employed were rigorous and could minimize bias and error in the findings). CB and RM met to compare their assessments, resolving all differences through discussion.

Using thematic synthesis methods [38-40], we explored themes concerning the characteristics of interventions, participants, and context acting as potential barriers and facilitators of delivery and receipt, and which themes applied across or only within the domains of sexual health, substance use, and mental health interventions. Synthesis followed a meta-ethnographic approach [41]. We undertook line-by-line coding of reports examining “first-order constructs” (directly quoted qualitative data) and second-order constructs (author interpretations). In the case of findings from quantitative study components, we coded author interpretations, first checking as part of quality assessment whether these aligned with quantitative data presented (ie, the extent to which study findings were grounded in the data, as noted above). Coding developed third-order constructs by drawing connections between these data. We did not exclude studies based on quality assessment, but rather gave less interpretive weight to conclusions that drew only on poorer-quality reports.

First, CB and RM prepared tables describing the quality of each evaluation, intervention details, study site and population, and pertinent findings. Second, the reviewers independently piloted coding of 2 high-quality studies. Coding began with in vivo codes which closely reflected the words used in the findings sections. The reviewers then grouped and organized codes, applying axial codes that reflected higher-order, cross-cutting themes. They then met to compare and contrast their coding, developing an overall set of codes. Next, they each went on to independently code the remaining reports, drawing on the agreed set of codes and developing new in vivo and axial codes as new themes emerged. At the end of this process, they met to compare their sets of codes. They identified commonalities, differences of emphasis, and contradictions to develop an overall analysis which drew on the strengths of the 2 sets of codes and which resolved any contradictions or inconsistencies.

Our search retrieved 26,044 unique results, including 1 identified via reference checking (see Figure 1). After title and abstract screening, 6 full texts were unobtainable and 345 reports were screened on full text. Of these, 37 reports were eligible for inclusion in the overall review. These reported on 28 unique studies and 23 unique interventions [42-78]. Reports were published between 2006 and 2020, with most published in 2015 or later.

Twelve reports were eligible for inclusion in the process evaluation synthesis (see Multimedia Appendix 2 for details of each intervention and Multimedia Appendix 3 for study characteristics) [42,49,52,54,59,63,64,67,68,75,76,78]. These reported on 11 studies which assessed 8 unique interventions, and 2 interventions were assessed in 2 different studies [54,59,67,78]. Included process evaluation reports presented findings on how intervention receipt (but not delivery) varied by characteristics of the intervention [42,49,52,59,63,64,67,68,75,76,78] participants [42,49,59,63,64,75,76,78] and context [54,75,78] but not providers. Additionally, 3 interventions addressed sexual health alone [54,68,75,78], 2 addressed mental health alone [42,63,64], 1 addressed sexual health and substance use [49,59,67], and 2 addressed all 3 outcomes of interest for this review [52,76] (see Multimedia Appendix 2). Moreover, 4 interventions targeted sexual minority youth or young adults [49,52,59,63,64,67,68], 2 targeted MSM more generally [42,75], 1 targeted rural MSM [54,78], and 1 targeted people living with HIV [76]. In terms of delivery mode, 5 were delivered via the internet [42,49,52,54,59,67,68,78], 2 via smartphone apps [75,76], and 1 via computer CD-ROM [63,64]. Process evaluations for 7 of the included interventions took place in the United States [42,49,52,54,59,67,68,75,76,78], and 1 took place in New Zealand [63,64,76].

Multimedia Appendix 3 shows the results of our quality assessments. In total, 11 of the 12 included reports were assessed as reporting findings that were grounded in the data presented. Overall quality varied, with most reports assessed as medium or high quality. In terms of the reliability or trustworthiness of their overall findings, 4 reports were assessed as medium quality [42,68,75,76] and 8 as high quality [49,52,54,59,63,64,67,78]. In terms of their overall usefulness for addressing our research questions, 4 were assessed as low quality [52,54,68,78], 3 as medium quality [63,64,67], and 5 as high quality [42,49,59,75,76]. Only 1 report was assessed as high quality in both reliability/trustworthiness and usefulness [49,59], and all were assessed as medium or high quality in at least one of these 2 domains [42,49,52,54,59,63,64,67,68,75,76,78].

Multimedia Appendix 4 shows the relationship between primary, secondary, and tertiary codes developed through our analysis and synthesis of process data.

One-third of reports included in the overall review included process evaluation data. All but one process evaluation took place in the United States. Most interventions targeted a single health domain of interest for this review (sexual health, substance use, or mental health), with the majority focused on sexual health. However, 2 aimed to address aspects of all 3 [52,76]. Some interventions employed personal tailoring, an approach that has been associated with effective eHealth behavior change interventions [79,80].

Process evaluations rarely explored how intervention receipt varied between contexts. We found no eligible reports examining what factors affected intervention delivery as opposed to receipt. This seems to reflect the emerging state of process evaluations in eHealth literature, with other reviews of eHealth interventions reporting a similar pattern [81-84]. There was some suggestion that slower internet speed could reduce acceptability of a multimedia intervention among rural MSM in the United States, who are less likely than nonrural residents to have high-speed internet at home [85].

In terms of intervention characteristics, as with use of eHealth interventions among general populations [83], participants appreciated when interventions were easy to use and free of technical problems, while incompatibility with mobile platforms detracted from acceptability and could impede participation. Privacy also emerged as an important aspect of acceptability, suggesting that detailed partner-level questions on sexual behavior could feel intrusive and that features protecting app access and obscuring the purpose of apps (for sensitive health domains) promote acceptability. The importance of privacy is also supported by existing evidence on behavior change interventions for MSM [86] and general populations [83].

Participants liked content that was interactive and aesthetically pleasing, and they enjoyed the use of diverse media such as animations, videos, and graphics. However, among rural MSM these media could also reduce loading times for users with slower internet connectivity. Although modular approaches could support users to absorb program content cumulatively, interventions that were too long detracted from acceptability, with some users preferring that less or no time be required between sessions. The ideal number and length of modules is likely dependent on a variety of participant, intervention, and contextual factors.

Individual tailoring based on participant characteristics and risk profiles increased acceptability, highlighting this as a particularly promising approach and aligning with other studies of eHealth behavioral interventions [79,84,87]. Participants valued when interventions presented scenarios and other content that reflected their experiences as MSM, an approach that stands in contrast to most existing eHealth interventions targeting mental health and HIV prevention [34,88]. Where interventions targeted sexual minority groups more broadly, some suggested further tailoring based on the sexual and gender identities of its users. The language and tone of intervention content emerged as an important factor shaping acceptability for MSM, who appreciated the use of colloquial, direct, “up-beat” [67], and sex-positive language. Our findings also highlight the importance of paying careful attention to language and framing to ensure that these affirm sexual-minority identities. That these concerns arose in interventions designed explicitly for sexual minority users, including one adapted for sexual minority young people using participatory approaches [63], suggests this is an important area to explore during the pilot phase of intervention development.

As with studies of eHealth interventions for general populations [81,83], perceived usefulness was key to acceptability. Participants liked gaining new knowledge and skills from eHealth interventions and developing an awareness of the relationship between sexual behaviors and substance use.

Although reviews of eHealth interventions for general populations report higher use and engagement among participants with higher levels of education [81,83,84], our findings suggest that in the context of generally high use of electronic devices among MSM [17], the targeting of intervention content might be a more important determinant of the relationship between education level and receipt of eHealth interventions than their electronic mode of delivery [49]. Similarly, our findings on the higher acceptability of the Keep it Up! intervention among Black, Latino, and other non-White users compared to White users suggest that eHealth interventions can be developed to enhance inclusive acceptability among racially and ethnically diverse users [49]. There was otherwise little evidence of engagement varying by sociodemographic factors, although findings on socioeconomic status were mixed [42,78]. Qualitative data suggest eHealth interventions can play a role in complementing external mental health support among MSM [63,76] and that interventions targeting all adolescents might struggle to pitch content appropriately for those across this age range [63].

Our process evaluation synthesis was limited by the size and quality of eligible reports. Most were assessed as medium- or high-quality in terms of their reliability and usefulness. However, studies often lacked depth and breadth of analysis, and only around half were judged to privilege MSM’s perspectives.

The vast majority of interventions targeted MSM only and all were evaluated principally among MSM, although 3 were assessed among samples that included cisgender women [63,64,68,76]. Author narratives and quantitative data did not always disaggregate MSM from other participants, presenting the possibility that specific findings from these 3 studies might reflect data from other groups. The process evaluation of Smartphone Self-Monitoring was the sole study contributing to findings on intervention benefits of self-monitoring and self-expression [76]. Although the intervention targeted people of all genders and sexual identities living with HIV, more than 80% of study participants identified as male and more than 80% identified as gay or bisexual [76]. In 2 studies, just under half of participants identified as female [63,64,68], but all themes informed by these studies also drew on other studies. The make-up of participants in these 3 studies is therefore unlikely to affect the validity of the themes to which they contributed. Studies of relevant interventions among broader sexual and gender minority populations might add further insight but could not be included, as we could not be certain which findings reflected experiences of or relevant to MSM.

eHealth interventions offer an avenue for MSM to access behavior change interventions privately, anonymously, and at times they find convenient. This synthesis identified several factors shaping MSM’s receipt of eHealth interventions addressing substance use, mental ill health, and sexual risk. Its findings suggest such interventions are acceptable for MSM across sociodemographic groups, although evidence in this area is limited and mixed. Different content for younger and older adolescents might be warranted. Variation in engagement and acceptability by participant characteristics should be explored in future research, and new interventions should be rigorously piloted to refine aspects affecting usability and acceptability [30,81].

Our review has identified several intervention characteristics affecting acceptability that existing research suggests are applicable to eHealth interventions for MSM and non-MSM populations alike. These include aspects of usability, length, aesthetics, multimedia use, and tailoring to participants’ personal and risk characteristics [79,81,83,84,86,87]. Other factors should be considered carefully in designing interventions for MSM, including ensuring that language and tone are affirming of sexual minority identity and that content reflects the reality and experiences of MSM. These findings can inform the development of integrated eHealth interventions to address the syndemic of substance use, mental ill health, and sexual risk among MSM and guide research questions for pilot and process evaluation studies. Going forward, process evaluations should explore a broader range of individual, intervention, and contextual factors that might affect implementation, and they should collect more in-depth—ideally qualitative—data privileging the perspectives of intended beneficiaries. Outcome evaluations of such eHealth interventions should conduct linked process evaluations wherever possible, which would shed further light on factors affecting how they are delivered and received [89].

Our findings regarding the value that participants place on interventions that address the reality of their lives and the interrelationships between the different domains of health suggest that eHealth interventions simultaneously addressing sexual health, substance use, and mental health might be particularly acceptable. Our review of theories of change [90] suggests that interventions addressing these different outcomes may aim to exert impacts via common mechanisms of action, further adding to the potential for eHealth interventions targeting multiple outcomes together. Our next analyses will assess the potential effectiveness of eHealth interventions on these outcomes.