Published on in Vol 22, No 5 (2020): May

Preprints (earlier versions) of this paper are available at, first published .
The Effectiveness of Electronic Health Interventions for Promoting HIV-Preventive Behaviors Among Men Who Have Sex With Men: Meta-Analysis Based on an Integrative Framework of Design and Implementation Features

The Effectiveness of Electronic Health Interventions for Promoting HIV-Preventive Behaviors Among Men Who Have Sex With Men: Meta-Analysis Based on an Integrative Framework of Design and Implementation Features

The Effectiveness of Electronic Health Interventions for Promoting HIV-Preventive Behaviors Among Men Who Have Sex With Men: Meta-Analysis Based on an Integrative Framework of Design and Implementation Features


1Jockey Club School of Public Health and Primary Care, The Chinese University of Hong Kong, Hong Kong, China (Hong Kong)

2TH Chan School of Public Health, Harvard University, Boston, MA, United States

3Center for Evidence Synthesis in Health, School of Public Health, Brown University, Providence, RI, United States

4School of Public Health, Shanghai University of Traditional Chinese Medicine, Shanghai, China

Corresponding Author:

Phoenix Kit-Han Mo, MSc, PhD, CPsychol

Jockey Club School of Public Health and Primary Care

The Chinese University of Hong Kong

Ngan Shing St, Shatin

Hong Kong

China (Hong Kong)

Phone: 852 22528765


Background: The disproportionately high prevalence of HIV among men who have sex with men (MSM) is a global concern. Despite the increasing utilization of electronic health (eHealth) technology in the delivery of HIV prevention interventions, few studies have systematically explored its effectiveness and association with various intervention characteristics.

Objective: This study aimed to conduct a meta-analysis of the effectiveness of eHealth technology–based interventions for promoting HIV-preventive behaviors among MSM and to determine effectiveness predictors within a framework integrating design and implementation features.

Methods: A systematic literature search using terms related to eHealth technology, HIV, the MSM population, and an experimental study design was performed using 5 databases (ie, MEDLINE, PsycINFO, EMBASE, Web of Science, and ProQuest Dissertations & Theses) and other sources (eg, bibliographies of relevant reviews and JMIR Publications). First, primary meta-analyses were conducted to estimate the effectiveness of eHealth interventions (d+) in changing 3 HIV-preventive behaviors among MSM: unprotected anal intercourse (UAI), HIV testing, and multiple sex partnership (MSP). Moderation analyses were then conducted to examine a priori effectiveness predictors including behavioral treatment components (eg, theory use, tailoring strategy use, navigation style, and treatment duration), eHealth technology components (eg, operation mode and modality type), and intervention adherence.

Results: A total of 46 studies were included. The overall effect sizes at end point were small but significant for all outcomes (UAI: d+=−.21, P<.001; HIV testing: d+=.38, P<.001; MSP: d+=−.26, P=.02). The intervention effects on UAI were significantly larger when compared with preintervention groups than with concurrent groups. Greater UAI reductions were associated with the increased use of tailoring strategies, provision of feedback, and tunneling navigation in interventions with a concurrent group, whereas reductions were associated with the use of self-paced navigation in interventions with a preintervention group. Greater uptake of HIV testing was associated with longer treatment duration; computer-mediated communication; and the use of messaging, social media, or a combined technology modality. Higher intervention adherence consistently predicted larger effects on UAI and HIV testing.

Conclusions: This study provided empirical evidence for the effectiveness of eHealth interventions in promoting HIV-preventive behaviors among MSM. Features of treatment content and eHealth technology might best predict the intervention effects on UAI and HIV testing, respectively. Most importantly, intervention adherence tended to play an important role in achieving better effectiveness. The findings could help inform the development of efficacious interventions for HIV prevention in the future.

J Med Internet Res 2020;22(5):e15977




In marked contrast to a declining trend in its global burden in the era of potent antiretroviral therapy, the HIV epidemic has continued to expand in the men who have sex with men (MSM) population across countries of all incomes in recent years [1,2]. The disproportionate burden is evidenced to be driven by a stable increase in HIV risk behaviors [3,4], which may offset the benefits of improved treatment coverage and biomedical advances [3,5]. Undiagnosed infections or infections diagnosed late arising from the relatively low uptake of HIV testing also fuel the epidemic by hampering treatment delivery and increasing transmission [6,7]. Therefore, the promotion of HIV-preventive behaviors is an indispensable part of comprehensive prevention efforts to control HIV [8]. Behavioral interventions among MSM have been found to significantly reduce risky sexual behaviors and increase HIV testing [9,10].

The global response to the ongoing epidemic warrants the application of innovative technology to develop efficacious behavioral interventions [7]. Electronic health (eHealth) is well recognized as the use of information and communication technologies for health, including websites, computerized programs or apps, social networking sites or chatrooms, email, or text messaging, that feature internet connectivity or digital interactivity via computer or mobile devices [11-13]. Previous quantitative reviews have shown that the impact of eHealth technology–based interventions on HIV prevention–related behaviors and theoretical correlates was significant but varied as a function of intervention characteristics [14,15].

New technology has been extensively used by MSM to socialize and seek sexual partners and to access information on sexual health [16,17]. An early meta-analysis revealed a significant effect of sexual health interventions delivered via interactive digital media for the overall population but not MSM, although a growing body of evidence has been accumulated since then [12]. Despite recent synthesis attempts focusing on MSM [18,19], the evaluation of effect magnitudes across outcomes and the disentanglement of efficacious components from complex intervention designs have been limited by the qualitative nature of the reviews. There is a lack of meta-analytic reviews that would allow rigorous testing of the a priori factors that predict intervention effectiveness [20] and thus inform the design and implementation of future programs.

Potential Predictors of Electronic Health Intervention Effectiveness

An eHealth technology–based behavioral intervention includes 2 elements: behavioral treatment (what type of intervention is designed and programmed to target behavior change) and eHealth technology (how the treatment is delivered via eHealth platforms). This study proposed a conceptualized framework to capture critical factors that are theoretically and empirically demonstrated to explain the effectiveness of eHealth interventions.

Behavioral Treatment Components

Treatment content is characterized by the use of theory for intervention development, as theoretical constructs can be operationalized into treatment techniques [21]. Significantly greater reductions in unprotected anal intercourse (UAI) among MSM have been observed for interventions reporting any use of theory [9], yet it is unclear whether the extent of use predicts intervention effectiveness, particularly in the eHealth context. A meta-analysis of internet-based behavioral interventions across health domains revealed a significant association between more extensive theory use and increased effect sizes [22].

Treatment content also features the application of tailoring strategies, defined as the integration of recipients’ responses into the intervention system to generate user-driven content [23]. There are 3 tailoring types: (1) feedback: providing unique recommendations derived from an assessment of individual needs or characteristics related to a given behavior; (2) adaptation: matching content to a relevant group based on known behavioral determinants; and (3) personalization: customizing content with personally identifiable information [24]. Computer technology–based interventions with individually tailored content adapted to the stage of change were shown to be more efficacious in increasing condom use in the general population [14].

The programming features of a treatment can strongly determine the intervention intensity. The amount of accessible content per interaction with the intervention differs by navigation style: tunneled interventions deliver treatment through a predetermined sequence of steps, whereas self-paced interventions release content all at once and permit recipients to control the navigation [25-27]. Existing evidence is inconclusive regarding the influence of navigation style on the effectiveness of eHealth interventions [26,28,29]. Treatment duration, referring to the time span of delivery, represents the overall intervention burden. Larger effects and higher adherence have been generally reported for online behavioral interventions of a shorter duration [23].

Electronic Health Technology Components

On the basis of the nature of users’ interaction with eHealth platforms, 2 operation modes of technology use for intervention delivery have been distinguished: (1) human-computer interaction (HCI), featuring direct interaction with a computerized system and automated delivery of a preprogrammed treatment and (2) computer-mediated communication (CMC), featuring remote delivery through interpersonal communications via eHealth media. These are considered to be inseparable, albeit essentially different aspects of eHealth technology and capable of deploying similar treatment strategies [25,30]. It has been reported that the provision of remote therapeutic support, not fully automated treatment, achieved significantly better psychological outcomes than passive control [31].

Four modality types have been applied to eHealth interventions for HIV prevention, and each of them possesses a unique capacity to facilitate intervention delivery [11,19,32]. The 2 most common modalities are the interactive module, in which users actively engage in an intervention following a preset workflow, and the static site, in which users passively receive prescriptive information [30]. A previous review suggested the superiority of interactive over static interventions in predicting increased condom use and sexually transmitted infection (STI) testing [12]. The strength of text messaging lies in the ubiquitous use of mobile devices to deliver real-time personalized interventions [11,33]. It can incorporate effective communication techniques and has demonstrated its efficacy in promoting health behavior change [33,34]. This study uses the broader term, “messaging,” to encompass related technologies that enable multimedia delivery [33]. Social media has emerged as a novel modality that features the creation and sharing of user-generated content in an online community [32,35], which can effectively promote HIV testing [35].

Exposure to Intervention Components

Intervention effectiveness observed in real-world practice can change with actual exposure to efficacious intervention components, and nonusage attrition is common in eHealth interventions [36]. Intervention adherence is conceptualized as the proportion of participants who engage in the intervention as prescribed to achieve a desired effect [29]. This allows for a comparable measurement of intervention exposure across use parameters, by contrasting the actual usage of an intervention during its implementation with the intended usage predefined at the design stage [25]. Adherence has been found to significantly predict the effectiveness of eHealth interventions [23].

This study aimed to conduct a meta-analysis of the effectiveness of eHealth technology–based interventions for promoting HIV-preventive behaviors among MSM and to provide an in-depth investigation of effectiveness predictors.

The guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses were followed [37].

Search and Selection

A systematic search was first performed in the following databases: MEDLINE, PsycINFO, EMBASE, Web of Science, and ProQuest Dissertations & Theses. The search strategy comprised 4 categories of terms related to eHealth technology, HIV or AIDS, the MSM population, and experimental study design, which were tailor-made for each database with restrictions to the English language and human studies (Multimedia Appendix 1). We manually searched bibliographies of relevant reviews, initially retrieved articles, and JMIR Publications to identify additional eligible studies. The eligibility criteria are listed in Table 1.

After removing the duplicates, 3657 relevant articles were identified from all the search sources. The title- and abstract-based eligibility screening was first conducted by 2 independent reviewers, which led to a selection of 143 articles for further full text–based assessment. Finally, 45 articles with a total sample size of 27,704 were deemed eligible and included in this meta-analysis [38-82]. The selection process is shown in Figure 1.

Table 1. Eligibility criteria for elements of a comprehensive search strategy.
ElementInclusion and exclusion criteria
  • Included: being exclusively or primarily (accounting for at least 80% of the sample) focused on MSMa or specifically conducting the efficacy evaluation among the MSM subgroup
  • Included: delivering interventions largely via eHealthb technologies, including internet-based tools or interactive computerized programs and administering behavior change interventions for the prevention of HIV infection
  • Excluded: solely using non-eHealth technologies (eg, telephone-based communications); not specifying the role of eHealth components in a multimedia program; or solely adopting biomedical strategies (eg, pre-exposure prophylaxis) or targeting the HIV care continuum after diagnosis (eg, treatment as prevention) or other sexual health topics (eg, contraception)
  • Included: using control conditions that are different from interventions in the eHealth components
  • Excluded: solely aimed at testing a group difference in other factors (eg, sample characteristics or treatment approaches)
  • Included: measuring HIV-preventive behaviors as efficacy outcomes and providing necessary statistics to estimate the effect size
  • Excluded: treating HIV-preventive behaviors as confounders or compensatory outcomes (eg, condom use measured in interventions on promoting monogamy); solely measuring behavioral correlates (eg, intentions) or biomarkers of engagement in HIV risk behaviors (eg, sexually transmitted infection occurrence); or computing a composite score for a group of behaviors
Study design
  • Included: a randomized controlled trial or nonrandomized experiment
  • Excluded: a single-group posttest-only experiment; an observational study; or a review or commentary

aMSM: men who have sex with men.

beHealth: electronic health.

Figure 1. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart of the screening process. eHealth: electronic health; MSM: men who have sex with men.
View this figure

Quality Assessment

In all, 44 intervention programs were involved (2 articles reported outcome measures at different follow-up points for the same program). Program quality was evaluated using the assessment tool of Schnall et al [19], which was developed based on the reporting standards for HIV intervention studies established by the HIV Prevention Research Synthesis Team of the US Centers for Disease Control and Prevention [83] and has been well applied in previous reviews [9,19]. An overall percentage score (0%-100%) was computed as the extent to which each program fulfilled the criteria across 7 categories: representativeness, bias and confounding, description of intervention, outcomes and follow-up, statistical analysis, strength of evidence, and group equivalence.

Data Extraction

Program Design

Data were extracted from each eligible program and its supplementary materials by one coder (MX) and double-checked by a second coder (AL). Information was first gathered on factors related to the program design, which were used to define the coding unit and to determine the grouping sets for the primary meta-analysis.

One intervention condition and one comparator were selected to form an independent study (a pairwise comparison) for the meta-analysis. If more than one intervention group was reported, each of them was separately included, given the potential variation in interventions, while the initial control group was proportionally divided up among intervention conditions to ensure independence [84]. If more than one control group was reported, the less intensive one was included for ease of interpretation [85]. If an eligible concurrent comparator was not obtained, or independent group contrasts were not the program focus, the preintervention status was treated as the comparator [86]. Finally, for each established comparison, identification items (eg, author) and sample characteristics (eg, age) were coded.

Study design was classified according to the outcome assessment points and comparison status (ie, using a concurrent or preintervention group). Comparator type was coded as (1) passive, including blank, waitlist, and attention controls or (2) active, including non-eHealth and lower tech eHealth treatment. Follow-up period was classified according to the interval between the intervention end and the follow-up points: (1) immediately postintervention; (2) short-term: ≤3 months; or (3) long-term follow-up: >3 months. The outcome measures with adequate studies for effect size pooling were UAI, condom use, HIV testing, and multiple sex partnership (MSP).

Intervention Features

A cluster of prespecified factors that captured the intervention characteristics were coded within the conceptualized framework described earlier. A relative coding rule was applied, that is, where a certain item was identical across the paired conditions, the study was coded as not having that feature, as it could not explain the conditional difference in the effectiveness.

With regard to the key features of treatment content, the extent of theory use was first assessed using the modified Theory Coding Scheme [21]. A total of 11 items pertinent to intervention development were used to compute an overall score [22], with a higher value indicating more extensive use of the theory. The second content feature was the use of tailoring strategies, which was coded in both ways, binary (whether adopting the feedback, adaptation, or personalization strategy) and continuous (the number of strategies adopted). The programming-specific treatment features were categorized into mutually exclusive groups: navigation style (self-paced or tunneled) and treatment duration (single session, ≤1 month, >1-3 months, and >3 months).

The utilization of eHealth technology was characterized by the operation mode (ie, HCI and CMC) and the modality type (ie, static site, interactive module, messaging, and social media). For both variables, interventions with the presence of more than one feature were grouped into a combined category to allow for exclusive coding. As indicators of intervention exposure, intended usage was first extracted and varying levels of actual usage were identified accordingly. Intervention adherence was then calculated as the percentage of participants whose actual usage matched the intended usage [25].

Data Synthesis

Effect Size Calculation

Standardized mean differences between conditions (Cohen d) and their standard error at each follow-up point were derived to represent the magnitude of intervention effects [87,88]. Following Morris and DeShon’s [89] procedures, the effect size estimate and its sampling error were computed or transformed to be scaled on a common “raw-score” metric, creating a synthesis across the study design. Unadjusted outcome measurements were retrieved to establish comparability across estimates. For studies in which means and standard deviations were not provided, effect measures reported in other forms (eg, risk ratio) were converted to the Cohen d statistic using well-developed calculators [90,91].

As recommended, UAI and condom use were combined into one outcome type, “UAI” [14]. For studies that used multiple instruments to measure an outcome, the most common instrument was chosen (eg, the frequency of UAI was prioritized over the count of UAI partners [56]). For studies measuring different subtypes of a certain outcome on the same scale (eg, serostatus-specific UAI [71]) or reporting subgroup effects by significant moderators that were unrelated to intervention components (eg, affect level [44]), all effect sizes were aggregated meta-analytically within the study. Consequently, each study yielded only one Cohen d value per outcome.


With an assumption of the intervention diversity, pooled effect sizes (d+) and 95% confidence intervals were generated using random effects models with the inverse variance weighting method. Values such as 0.2, 0.5, and 0.8 were interpreted as small, moderate, and large, respectively [92]. Primary meta-analyses were performed to combine effect sizes within different study sets for each outcome: (1) an overall estimate at the end point (one effect arising per study); (2) group-specific estimates of the end point effect by comparison status (nested groupings); and (3) group-specific estimates by follow-up period (a multi-wave study contributing multiple effects). To maximize analytical power, the first type of estimation based on a full data set was used for further analyses.

Heterogeneity across studies was assessed by using the Q statistic with a P value <.05, indicating the presence of significant heterogeneity, and quantified by using the I2 statistic, where a value of 30% to 60% denoted “moderate” and ≥75% denoted “considerable” heterogeneity [93]. Wherever considerable heterogeneity was found, outliers were identified as studies that significantly distorted the pooled effect using influence analyses [94] and were removed to ensure the accuracy and generalizability of the findings. Publication bias was then assessed by visually inspecting the funnel plot of effect sizes and conducting an Egger regression test to examine the plot asymmetry when there were at least 10 estimates [95].

Next, secondary analyses were performed to test the moderation effect of intervention features where adequate studies were available (n≥10) [96]. Subgroup and meta-regression analyses were applied for dichotomous and continuous measures, respectively; a significant moderator was indicated by a P value <.05 for the heterogeneity across subgroups (Qb) or the regression coefficient (β). All analyses were conducted in R 3.5.2 with the metafor package (Wolfgang Viechtbauer) [97].

Descriptions of Program Characteristics

Over half of the eligible programs (23/44, 52%) were conducted in the United States, and 10 and 8 programs were conducted in Asia and Europe, respectively. Among programs reporting the respective background characteristics, most of the samples had a mean age below 30 years (24/44, 54%), were multiracial or Asian (29/39, 74%), were mainly (≥70%) composed of homosexual males or gays (22/28, 78%), and were non-HIV positive or of mixed (both positive and negative) status (33/36, 92%). In addition, the median overall quality score was 84.2%; the assessment results are provided in detail in Multimedia Appendix 2. A total of 46 studies were further identified from these programs; most of them conducted a concurrent comparison (n=29) and used a passive comparator (n=39). The study characteristics are provided in detail in Multimedia Appendix 3.

Results of Primary Analyses

Unprotected Anal Intercourse

In combing all 35 studies that tested for UAI, a significant and small overall intervention effect was observed at end point (d+=−.32; P<.001), albeit with considerable heterogeneity (I2=83.3%). One study included subjects who reported recent condom-less sex with partners of either gender before enrollment; such a high-risk preintervention status might have contributed to the greatly decreased UAI (d=−1.13; SE 0.21) [49]. Moreover, the sample had the highest mean age of 45.1 years among all studies. Another study was focused on MSM sex workers and primarily addressed the context of transactional sex [63]. An extremely positive effect was found for reduced UAI with nonpaying male partners (d=−3.95; SE 0.35). Removing both outliers resulted in a smaller effect with moderate heterogeneity (d+=−.21; P<.001; I2=50.8%).

When stratified by the partner-specific outcome, a larger end point d+ was shown for UAI with main partners than with nonmain partners. When stratified by the follow-up period, a significant d+ was obtained at all points. The nested subgroup analysis found a significantly greater intervention effect when compared with preintervention groups than with concurrent groups (Qb=13.38; P<.001); hence, the meta-analysis was performed separately for the 2 types of comparison status as recommended (Figures 2 and 3) [89].

Figure 2. A forest plot of intervention effects on unprotected anal intercourse against concurrent comparison groups.
View this figure
Figure 3. A forest plot of intervention effects on unprotected anal intercourse against preintervention comparison groups.
View this figure
HIV Testing

Similarly, pooling the studies that tested for HIV testing revealed a significant, small overall effect at endpoint (n=23; d+=−.32; P<.001). The considerable heterogeneity (I2=84.0%) identified was largely attributable to the detection of outliers. Bourne et al [40] tested a texting reminder intervention against a blank control group involving participants who had refused such reminders; thus, the extreme effect size (d=.80; SE 0.06) could have been susceptible to volunteer bias. Wang et al [79] promoted a novel self-testing approach and distributed free home-based testing kits; the uptake of any type of testing was found to be considerably higher in the intervention group (d=1.18; SE 0.14). Mikolajczak et al [62] unexpectedly found a negative but nonsignificant intervention effect on HIV or STI testing relative to an active comparator (d=−.13; SE 0.12). The exclusion of outliers caused a slight change in the effectiveness but substantially reduced heterogeneity (d+=.38; P<.001; I2=64.2%; Figure 4). A larger d+ was further shown at postintervention than at short-term follow-up; however, only one sample was followed up for more than 3 months. No significant group difference by comparison status was observed.

Figure 4. A forest plot of intervention effects on HIV testing.
View this figure
Multiple Sex Partnership

A significant, small overall effect was observed on the reduced MSP at endpoint with moderate heterogeneity (d+=−.26; P=.02; n=6; I2=60.2%; Figure 5). The largest d+ was shown at medium-term follow-up among all follow-up periods, although only a few studies were available for each grouping. No between-group test was performed for comparison status because of the small number of studies. Publication bias was detected for none of the outcomes. More results of primary analyses are presented in Table 2.

Figure 5. A forest plot of intervention effects on multiple sex partnership.
View this figure
Table 2. Overall effect sizes and stratification by study design features.
StudiesPopulation, nCohen d (95% CI)P valueQw (P value)aI2 (%)Qb (P valueb)

All studies at endpoint35−.32 (−.43 to −.20)<.001203.32 (<.001)83.3

Studies at endpoint without outliers33−.21 (−.28 to −.14)<.00165.08 (<.001)50.8

Partner-specific UAIe,f,gN/A

With main partners6−.42 (−.57 to −.27)<.0015.18 (.39)3.4

With nonmain partners9−.20 (−.27 to −.13)<.0017.78 (.46)0.0

Follow-up periodfN/A

Postintervention16−.22 (−.36 to −.08).00148.02 (<.001)68.8

≤3 monthse19−.21 (−.28 to −.12)<.00135.92 (.007)49.9

>3 monthse6−.26 (−.43 to −.08).00411.60 (.04)56.9

Comparison statuse,g13.38 (<.001)

Concurrent19−.10 (−.18 to −.03).00424.29 (.14)25.9

Preintervention14−.31 (−.40 to −.23)<.00118.16 (.15)28.4
HIV testing

All studies at end point23.41 (.28 to .54)<.001137.43 (<.001)84.0

Studies at end point without outliers20.38 (.27 to .48)<.00153.04 (<.001)64.2

Follow-up periodfN/A

Postintervention15.46 (.38 to .55)<.00123.49 (.05)40.4

≤3 monthse8.24 (.09 to .40).00212.57 (.08)44.3

>3 months1.47 (.25 to .69)<.001N/AN/A

Comparison statuse,g0.42 (.52)

Concurrent11.39 (.25 to .53)<.00136.67 (<.001)72.7

Preintervention9.33 (.22 to .44)<.0019.14 (.33)12.4
Multiple sex partnership

All studies at endpoint6−.26 (−.48 to −.05).0212.56 (.03)60.2

Follow-up periodfN/A

Postintervention2−.18 (−.33 to −.03).020.08 (.78)0.0

≤3 months3−.19 (−.75 to .37).5011.65 (.003)82.8

>3 months2−.36 (−.56 to −.16)<.0010.34 (.56)0.0

aQw denotes the degree of within-group heterogeneity.

bQb denotes the degree of between-group difference in the pooled effect sizes.

cUAI: unprotected anal intercourse.

dN/A: not applicable.

eStudies with exclusion of outliers.

fNon-nested groupings: one study could contribute to more than one grouping.

gEffect sizes at the endpoint were pooled.

Results of Moderation Analyses

Behavioral Treatment Components

Most of the studies were designed based on a theory (n=37); the most common one was the information-motivation-behavioral skills model (IMB; n=14). The intervention effects on UAI and HIV testing appeared to be comparable regardless of whether any theory was used. Among the theory-based studies, IMB was significantly associated with greater UAI reductions than other theories for concurrent comparisons (Qb=4.33; P=.04). The more extensive use of tailoring strategies was significantly associated with decreased UAI for concurrent comparisons (β=−.08; P=.01). Specifically, feedback was the only strategy that had a significantly negative effect on UAI (Qb=4.00; P=.04), whereas adopting the personalization strategy tended to have a positive effect on HIV testing (Qb=2.76; P=.10).

Significantly greater UAI reductions were shown for the tunneled (Qb=7.23; P=.01) and self-paced (Qb=4.23; P=.04) treatments relative to concurrent and preintervention groups, respectively. Such a moderating effect of navigation style was not detected for HIV testing. Longer treatments showed a significantly greater increase in HIV testing than those with a single session or those lasting ≤1 month (Qb=16.97; P<.001). However, the effects on UAI did not significantly differ across treatment durations.

Electronic Health Technology Components

Over half of the interventions were operated via HCI (n=26), and the others were in a CMC (n=10) or combined (n=8) mode. More studies used a single modality (n=32) than those incorporating multi-modalities in the same intervention albeit with varying combination patterns (n=14). No between-group difference was observed in the intervention effects on UAI by operation mode or modality type. The CMC and combined modes tended to present a larger d+ value for HIV testing than did the HCI mode (Qb=2.76; P=.09). When separately exploring the effect of specific modes, the use of CMC predicted a significantly greater increase in HIV testing (Qb=4.38; P=.04), whereas HCI use showed no association with effectiveness. Moreover, a significantly larger d+ value was found for the use of messaging, social media, and multi-modalities than for static sites (Qb=12.79; P=.005).

Intervention Adherence

Among all interventions, the adherence rates ranged from 25.5% to 100%. A higher adherence rate was significantly associated with a decrease in UAI for concurrent comparisons (β=−.27; P=.03). Given the highly negative skewed distribution, the studies were evenly divided into 3 levels to reflect the degree of relative adherence; the adherence rates were above 91.6% in the first tertile and below 76% in the third tertile. Intervention effects on HIV testing significantly differed by adherence level (Qb=7.28; P=.03). A moderate d+ value was observed for high-level studies (in the first tertile) and a small one for mid- and low-level studies. Details of the intervention features are presented in Multimedia Appendix 4; more results of moderation analyses are presented in Multimedia Appendix 5.

Principal Findings

This study identified 46 eligible studies published since 2006, which highlight the increasing application of eHealth technology in intervention delivery for HIV prevention over the last decade. Most studies tested the effectiveness of an entire intervention against a passive comparator. Those with an active comparator delivered a treatment that was either a basic component of a comprehensive intervention package [79] or existing online information that the intervention group might be exposed to as well [62]; hence, it is possible to isolate the effects of specific intervention components.

Effectiveness of Electronic Health–Based Interventions

The primary meta-analysis consistently revealed a significant and small overall intervention effect for all the behavioral outcomes at endpoint. Considerable heterogeneity was nevertheless detected for UAI and HIV testing, and some influential cases were further identified to show extreme effect sizes and distinctive study characteristics. Removal of the outliers weakened the pooled effectiveness for both outcomes to some extent. In line with extant reviews, eHealth-based interventions exerted a greater impact on HIV testing [12] and the number of sex partners [14] than on condom use. Notably, the magnitude of effectiveness observed is seemingly higher than that in the general population [12], which suggests that the MSM community may benefit more than others from eHealth-based interventions.

Furthermore, the meta-analysis within the groupings by outcome type demonstrated greater reductions in UAI with main partners than with nonmain partners. It is recommended that future interventions target the partner-specific determinants of risky sexual behaviors. Most of the studies evaluated the postintervention and short-term effects across outcomes. Only 6 study samples were followed up for more than 3 months and showed a comparable decrease in UAI to that at earlier timepoints. It was not possible to estimate the long-term effect on HIV testing and MSP owing to the lack of data. This finding emphasizes the need to explore the effectiveness of eHealth interventions in maintaining behavior changes, especially given the necessity for consistent condom use and regular HIV testing [7,98]. The greater effect on UAI found for the pre-post comparison was probably confounded by factors unrelated to the intervention (eg, fatigue). Nevertheless, the heterogeneity between comparison types may also have resulted from the different intervention components deployed by the 2 groups of studies, as has been discussed in the following sections.

Predictors of Intervention Effectiveness

Behavioral Treatment Components

This study confirmed the previous finding [14] that the use of the theory did not moderate the effectiveness of eHealth-based interventions for HIV prevention, although the majority of treatments were developed on a theoretical basis. The overall extent of theory use failed to significantly influence effectiveness across outcomes. However, this finding may merely reflect a lack of sensitivity in distinguishing an effect [99]. The number of tailoring strategies used significantly moderated effectiveness in reducing UAI, and only the most frequently used strategy, that is, feedback, was further shown to be effective. Treatments were less tailored for interventions promoting HIV testing, and only the moderating effect of personalization use reached marginal significance. Some evidence supporting the superiority of certain strategies has been reported [23]. The use of feedback and personalization commonly featured tailoring at an individual level, whereas the use of adaptation generated content matched to group-level factors.

Inconclusive results were found for the role of navigation style. Tunneled treatments achieved greater UAI reductions than self-paced treatments in studies using a concurrent group but lower reductions in those using a preintervention group. A plausible explanation is that the tunneling pattern differed across comparison types. A tunneled treatment by definition requires multiple interactions with the intervention, which could impair the effectiveness of eHealth behavioral interventions [23]. Over 60% of treatments with a preintervention group involved more than 5 modules, whereas only 24% of those with a concurrent group did so. Above all, the efficacy of the tunneling design itself is controversial as, on the one hand, the sequential release could ameliorate information anxiety and enhance the behavioral change process, whereas, on the other hand, the artificial confines could inhibit typical information-foraging behavior and intervention participants may lack the motivation to accommodate such constraints [100,101].

Longer treatments significantly predicted higher uptake of HIV testing but not reduced UAI, which adds to the evidence in favor of a difference in the influence of treatment duration on sexual and detection behaviors [12]. However, this finding might merely reject a linear relationship between treatment duration and the effect on UAI. A recent meta-analysis even demonstrated a negative effect of increased intervention length on intervention adherence and behavioral impacts, driven by decreased motivation over time [23].

Electronic Health Technology Components

Taking advantage of both human communication (eg, flexibility and rapport building [49]) and eHealth-enabled capacities (eg, convenience and anonymity [30]), the use of CMC was shown to be effective in increasing HIV testing. There was little evidence for the moderating effect of the operation mode on UAI, although only 1 intervention used a CMC mode. The results illustrate the need to incorporate human involvement into eHealth interventions. A variety of communicative functions have proved their efficacy in promoting behavior change, including counselor- or user-initiated conversations and peer-to-peer interactions [22]. Moreover, the greatest effectiveness was obtained with a combined mode, so the relevant question is not whether HCI or CMC is superior but, rather, how to combine them to maximize persuasiveness.

Similarly, intervention effects on the outcome of HIV testing but not UAI varied by modality type. Messaging, social media, and the combined type achieved a comparably small-to-medium effect. Messaging, largely operated via portable devices, affords individualized interventions (eg, location-based services) and dynamic connections (eg, response on request) [11,102]. Social media stands out for its capabilities to influence social norms and create a peer-supportive environment [103], especially within a closely connected community such as MSM [35]. It also enables interventions to blend into users’ daily lives by utilizing well-established platforms (eg, Facebook) [32]. The use of hybrid modalities may further generate a synergistic effect.

Intervention Adherence

Consistent with previous literature [23], a high degree of adherence to eHealth interventions predicted a protective effect on UAI and HIV testing. Assuming intention-to-treat as the motive to engage in an intervention, adherence is rooted in the properties of intervention design. Efficacious strategies (eg, self-monitoring), observable benefits (eg, health outcomes), and feasible programming (eg, appropriate workload) can all bolster engagement [26,36]. Usability of eHealth technology is also critical to minimizing discontinuance [36].


This study focused on the behavioral aspects of HIV prevention. Notably, it did not indicate the superiority of stand-alone behavioral interventions over other prevention tools (eg, pre-exposure prophylaxis) but was, rather, an attempt to explore the possibility of leveraging technological advances to strengthen an integral part of comprehensive biobehavioral prevention efforts, especially at a transition stage when the role of biomedical prevention is limited by its slow and uneven scale-up [7,104]. It is also impossible to synthesize evidence for other preventive behaviors owing to insufficient data (eg, serosorting) or highly heterogeneous measurements (eg, substance use). The potential interplay among intervention components has not been examined because of the lack of statistical power. Some other effectiveness predictors may also be missed (eg, eHealth literacy [26]), although the key factors with recognized terminology and accessible coding sources were selected. Finally, these findings could not represent studies reported in a language other than English, although there was no evidence of biased effectiveness among previous language-restrictive meta-analyses [105].


This meta-analysis demonstrated eHealth technology to be a promising tool for delivering HIV prevention interventions among the MSM population. Nevertheless, there is limited evidence on the long-term impact of such interventions, and more research is warranted to investigate their application in non-Western contexts. Our findings suggested that enhanced behavioral treatment (eg, the use of tailoring strategies) determined the effect of eHealth interventions on UAI reductions, whereas appropriate programming (eg, longer treatment duration) and advanced eHealth technology (eg, the use of social media) predicted an increase in HIV testing. Future intervention design should focus on integrating efficacious treatment and technology components as well as on exploring their potential interplay. It is also recommended to incorporate implementation strategies to improve intervention adherence and, thus, achieve better effectiveness.


The authors would like to express their heartfelt gratitude to Ms Beizhu Ye and Ms Mei Zhao, who provided support with screening, and to the authors of the original studies, who provided additional information necessary for this review. This study was supported by the Early Career Scheme for 2017-2018 (reference no. 24106317), and the funding agent was the University Grants Committee of Hong Kong. The funding source(s) was not involved in the literature review, systematic review, meta-analysis, and writing of the report.

Conflicts of Interest

None declared.

Multimedia Appendix 1

The search strategy.

DOCX File , 43 KB

Multimedia Appendix 2

Results of quality assessment.

DOCX File , 33 KB

Multimedia Appendix 3

Characteristics of 46 studies included in the present meta-analysis.

DOCX File , 153 KB

Multimedia Appendix 4

Design and implementation features of included interventions.

DOCX File , 156 KB

Multimedia Appendix 5

Results of moderation analyses.

DOCX File , 36 KB

  1. Beyrer C, Baral SD, van Griensven F, Goodreau SM, Chariyalertsak S, Wirtz AL, et al. Global epidemiology of HIV infection in men who have sex with men. Lancet 2012 Jul 28;380(9839):367-377 [FREE Full text] [CrossRef] [Medline]
  2. Beyrer C, Sullivan P, Sanchez J, Baral SD, Collins C, Wirtz AL, et al. The increase in global HIV epidemics in MSM. AIDS 2013 Nov 13;27(17):2665-2678. [CrossRef] [Medline]
  3. Hess KL, Crepaz N, Rose C, Purcell D, Paz-Bailey G. Trends in sexual behavior among men who have sex with men (MSM) in high-income countries, 1990-2013: a systematic review. AIDS Behav 2017 Oct;21(10):2811-2834 [FREE Full text] [CrossRef] [Medline]
  4. van Griensven F, de Lind van Wijngaarden JW. A review of the epidemiology of HIV infection and prevention responses among MSM in Asia. AIDS 2010 Sep;24(Suppl 3):S30-S40. [CrossRef] [Medline]
  5. Mei S, Quax R, van de Vijver D, Zhu Y, Sloot PM. Increasing risk behaviour can outweigh the benefits of antiretroviral drug treatment on the HIV incidence among men-having-sex-with-men in Amsterdam. BMC Infect Dis 2011 May 11;11:118 [FREE Full text] [CrossRef] [Medline]
  6. Campbell CK, Lippman SA, Moss N, Lightfoot M. Strategies to increase HIV testing among MSM: a synthesis of the literature. AIDS Behav 2018 Aug;22(8):2387-2412. [CrossRef] [Medline]
  7. Beyrer C, Baral SD, Collins C, Richardson ET, Sullivan PS, Sanchez J, et al. The global response to HIV in men who have sex with men. Lancet 2016 Jul 9;388(10040):198-206. [CrossRef] [Medline]
  8. Higa DH, Crepaz N, Marshall KJ, Kay L, Vosburgh HW, Spikes P, et al. A systematic review to identify challenges of demonstrating efficacy of HIV behavioral interventions for gay, bisexual, and other men who have sex with men (MSM). AIDS Behav 2013 May;17(4):1231-1244 [FREE Full text] [CrossRef] [Medline]
  9. Herbst JH, Sherba RT, Crepaz N, Deluca JB, Zohrabyan L, Stall RD, HIV/AIDS Prevention Research Synthesis Team. A meta-analytic review of HIV behavioral interventions for reducing sexual risk behavior of men who have sex with men. J Acquir Immune Defic Syndr 2005 Jun 1;39(2):228-241. [Medline]
  10. Lu H, Liu Y, Dahiya K, Qian H, Fan W, Zhang L, et al. Effectiveness of HIV risk reduction interventions among men who have sex with men in China: a systematic review and meta-analysis. PLoS One 2013;8(8):e72747 [FREE Full text] [CrossRef] [Medline]
  11. Noar SM, Willoughby JF. eHealth interventions for HIV prevention. AIDS Care 2012;24(8):945-952 [FREE Full text] [CrossRef] [Medline]
  12. Swanton R, Allom V, Mullan B. A meta-analysis of the effect of new-media interventions on sexual-health behaviours. Sex Transm Infect 2015 Feb;91(1):14-20. [CrossRef] [Medline]
  13. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]
  14. Noar SM, Black HG, Pierce LB. Efficacy of computer technology-based HIV prevention interventions: a meta-analysis. AIDS 2009 Jan 2;23(1):107-115. [CrossRef] [Medline]
  15. Noar SM, Pierce LB, Black HG. Can computer-mediated interventions change theoretical mediators of safer sex? A meta-analysis. Hum Commun Res 2010;36(3):261-297. [CrossRef]
  16. Holloway IW, Rice E, Gibbs J, Winetrobe H, Dunlap S, Rhoades H. Acceptability of smartphone application-based HIV prevention among young men who have sex with men. AIDS Behav 2014 Feb;18(2):285-296 [FREE Full text] [CrossRef] [Medline]
  17. Zou HC, Fan S. Characteristics of men who have sex with men who use smartphone geosocial networking applications and implications for HIV interventions: a systematic review and meta-analysis. Arch Sex Behav 2017 May;46(4):885-894. [CrossRef] [Medline]
  18. Knight R, Karamouzian M, Salway T, Gilbert M, Shoveller J. Online interventions to address HIV and other sexually transmitted and blood-borne infections among young gay, bisexual and other men who have sex with men: a systematic review. J Int AIDS Soc 2017 Nov;20(3):- [FREE Full text] [CrossRef] [Medline]
  19. Schnall R, Travers J, Rojas M, Carballo-Diéguez A. eHealth interventions for HIV prevention in high-risk men who have sex with men: a systematic review. J Med Internet Res 2014 May 26;16(5):e134 [FREE Full text] [CrossRef] [Medline]
  20. Deeks JJ, Higgins JP, Altman DG. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0.: The Cochrane Collaboration; 2011. Specify Characteristics in Advance   URL: [accessed 2018-09-01]
  21. Michie S, Prestwich A. Are interventions theory-based? Development of a theory coding scheme. Health Psychol 2010 Jan;29(1):1-8. [CrossRef] [Medline]
  22. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
  23. Cugelman B, Thelwall M, Dawes P. Online interventions for social marketing health behavior change campaigns: a meta-analysis of psychological architectures and adherence factors. J Med Internet Res 2011 Feb 14;13(1):e17 [FREE Full text] [CrossRef] [Medline]
  24. Lustria ML, Cortese J, Noar SM, Glueckauf RL. Computer-tailored health interventions delivered over the Web: review and analysis of key components. Patient Educ Couns 2009 Feb;74(2):156-173. [CrossRef] [Medline]
  25. Kelders SM, Kok RN, Ossebaard HC, van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
  26. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2017 Jun;7(2):254-267 [FREE Full text] [CrossRef] [Medline]
  27. Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res 2014 Jun 5;16(6):e146 [FREE Full text] [CrossRef] [Medline]
  28. Wahle F, Bollhalder L, Kowatsch T, Fleisch E. Toward the design of evidence-based mental health information systems for people with depression: a systematic literature review and meta-analysis. J Med Internet Res 2017 May 31;19(5):e191 [FREE Full text] [CrossRef] [Medline]
  29. Wildeboer G, Kelders SM, van Gemert-Pijnen JE. The relationship between persuasive technology principles, adherence and effect of web-Based interventions for mental health: A meta-analysis. Int J Med Inform 2016 Dec;96:71-85. [CrossRef] [Medline]
  30. Barak A, Hen L, Boniel-Nissim M, Shapira N. A comprehensive review and a meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. J Technol Hum Serv 2008;26(2-4):109-160. [CrossRef]
  31. Kuester A, Niemeyer H, Knaevelsrud C. Internet-based interventions for posttraumatic stress: a meta-analysis of randomized controlled trials. Clin Psychol Rev 2016 Feb;43:1-16. [CrossRef] [Medline]
  32. Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, internet and web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep 2015 Mar;12(1):173-190 [FREE Full text] [CrossRef] [Medline]
  33. Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health 2015 Mar 18;36:393-415 [FREE Full text] [CrossRef] [Medline]
  34. Head KJ, Noar SM, Iannarino NT, Harrington NG. Efficacy of text messaging-based interventions for health promotion: a meta-analysis. Soc Sci Med 2013 Nov;97:41-48. [CrossRef] [Medline]
  35. Cao B, Gupta S, Wang J, Hightow-Weidman LB, Muessig KE, Tang W, et al. Social media interventions to promote HIV testing, linkage, adherence, and retention: systematic review and meta-analysis. J Med Internet Res 2017 Nov 24;19(11):e394 [FREE Full text] [CrossRef] [Medline]
  36. Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [CrossRef] [Medline]
  37. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 2009 Jul 21;6(7):e1000100 [FREE Full text] [CrossRef] [Medline]
  38. Anand T, Nitpolprasert C, Jantarapakde J, Meksena R, Phomthong S, Phoseeta P, et al. Implementation and impact of a technology-based HIV risk-reduction intervention among Thai men who have sex with men using 'Vialogues': a randomized controlled trial. AIDS Care 2020 Mar;32(3):394-405. [CrossRef] [Medline]
  39. Bauermeister JA, Pingel ES, Jadwin-Cakmak L, Harper GW, Horvath K, Weiss G, et al. Acceptability and preliminary efficacy of a tailored online HIV/STI testing intervention for young men who have sex with men: the Get Connected! program. AIDS Behav 2015 Oct;19(10):1860-1874 [FREE Full text] [CrossRef] [Medline]
  40. Bourne C, Knight V, Guy R, Wand H, Lu H, McNulty A. Short message service reminder intervention doubles sexually transmitted infection/HIV re-testing rates among men who have sex with men. Sex Transm Infect 2011 Apr;87(3):229-231. [CrossRef] [Medline]
  41. Bowen AM, Williams ML, Daniel CM, Clayton S. Internet based HIV prevention research targeting rural MSM: feasibility, acceptability, and preliminary efficacy. J Behav Med 2008 Dec;31(6):463-477 [FREE Full text] [CrossRef] [Medline]
  42. Carpenter KM, Stoner SA, Mikko AN, Dhanak LP, Parsons JT. Efficacy of a web-based intervention to reduce sexual risk in men who have sex with men. AIDS Behav 2010 Jun;14(3):549-557 [FREE Full text] [CrossRef] [Medline]
  43. Chiasson MA, Shaw FS, Humberstone M, Hirshfield S, Hartel D. Increased HIV disclosure three months after an online video intervention for men who have sex with men (MSM). AIDS Care 2009 Sep;21(9):1081-1089. [CrossRef] [Medline]
  44. Christensen JL. CORE – Aggregating the world's open access research papers. Ann Arbor: University of Southern California; 2007. When It's Good to Feel Bad: How Responses to Virtual Environments Predict Real-Life Sexual Risk-reduction   URL: [accessed 2018-09-01]
  45. Christensen JL, Miller LC, Appleby PR, Corsbie-Massay C, Godoy CG, Marsella SC, et al. Reducing shame in a game that predicts HIV risk reduction for young adult MSM: a randomized trial delivered nationally over the Web. J Int AIDS Soc 2013 Nov 13;16(3 Suppl 2):18716 [FREE Full text] [CrossRef] [Medline]
  46. Daniel C, Bowen A, Williams M, Clayton S, Ross L. Assessment of long-term efficacy of an internet delivered HIV risk reduction intervention for rural MSM. In: Abstracts of the 29th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine.: Annals of behavioral medicine; 2008 Presented at: Behavioral medicine: celebrating the past, inspiring the future; March 26-29, 2008; San Diego p. s105   URL:
  47. Davidovich U, de Wit JBF, Stroebe W. Using the Internet to reduce risk of HIV-infection in steady relationships: a randomized controlled trial of a tailored intervention for gay men. In: Davidovich U, editor. Liaisons Dangereuses: HIV Risk Behavior and Prevention in Steady Gay Relationships. Amsterdam: Roel & Uitgeefprojecten; 2006:96-122.
  48. Desai M, Nardone A, Burns F, Mercey D, Gilson R, Muniina P, et al. Active recall of men who have sex with men (MSM) for an HIV/STI testing: a feasible and effective strategy? In: Abstracts of the Third Joint Conference of the British HIV Association (BHIVA) with the British Association for Sexual Health and HIV (BASHH). 2014 Presented at: Third Joint Conference of the BHIVA and the BASHH; April 1-4, 2014; Liverpool p. 109   URL:
  49. Fernandez MI, Hosek SG, Hotton AL, Gaylord SE, Hernandez N, Alfonso SV, et al. A Randomized Controlled Trial of POWER: an internet-based HIV prevention intervention for black bisexual men. AIDS Behav 2016 Sep;20(9):1951-1960. [CrossRef] [Medline]
  50. Greene GJ, Madkins K, Andrews K, Dispenza J, Mustanski B. Implementation and Evaluation of the Keep It Up! Online HIV Prevention Intervention in a Community-Based Setting. AIDS Educ Prev 2016 Jun;28(3):231-245. [CrossRef] [Medline]
  51. Habarta N, Boudewyns V, Badal H, Johnston J, Uhrig J, Green D, et al. CDC'S Testing Makes Us Stronger (TMUS) Campaign: Was Campaign Exposure Associated With HIV Testing Behavior Among Black Gay and Bisexual Men? AIDS Educ Prev 2017 Jun;29(3):228-240. [CrossRef] [Medline]
  52. Hightow-Weidman LB, Pike E, Fowler B, Matthews DM, Kibe J, McCoy R, et al. feasibility and acceptability of delivering an internet intervention to young Black men who have sex with men. AIDS Care 2012;24(7):910-920 [FREE Full text] [CrossRef] [Medline]
  53. Hilliam A, Fraser L, Turner L. NHS Health Scotland. Scotland: NHS Health Scotland; 2011 Feb. HIV Wake-Up Campaign Evaluation   URL: [accessed 2018-09-01]
  54. Hirshfield S, Chiasson MA, Joseph H, Scheinmann R, Johnson WD, Remien RH, et al. An online randomized controlled trial evaluating HIV prevention digital media interventions for men who have sex with men. PLoS One 2012;7(10):e46252 [FREE Full text] [CrossRef] [Medline]
  55. Kasatpibal N, Viseskul N, Srikantha W, Fongkaew W, Surapagdee N, Grimes RM. Effects of internet-based instruction on HIV-prevention knowledge and practices among men who have sex with men. Nurs Health Sci 2014 Dec;16(4):514-520 [FREE Full text] [CrossRef] [Medline]
  56. Ko N, Hsieh C, Wang M, Lee C, Chen C, Chung A, et al. Effects of Internet popular opinion leaders (iPOL) among internet-using men who have sex with men. J Med Internet Res 2013 Feb 25;15(2):e40 [FREE Full text] [CrossRef] [Medline]
  57. Lau JT, Lau M, Cheung A, Tsui HY. A randomized controlled study to evaluate the efficacy of an internet-based intervention in reducing HIV risk behaviors among men who have sex with men in Hong Kong. AIDS Care 2008 Aug;20(7):820-828. [CrossRef] [Medline]
  58. Lau JTF, Lee AL, Tse WS, Mo PK, Fong F, Wang Z, et al. A randomized control trial for evaluating efficacies of two online cognitive interventions with and without fear-appeal imagery approaches in preventing unprotected anal sex among Chinese men who have sex with men. AIDS Behav 2016 Sep;20(9):1851-1862. [CrossRef] [Medline]
  59. Leluțiu-Weinberger C, Manu M, Ionescu F, Dogaru B, Kovacs T, Dorobănțescu C, et al. An mHealth intervention to improve young gay and bisexual men's sexual, behavioral, and mental health in a structurally stigmatizing national context. JMIR Mhealth Uhealth 2018 Nov 14;6(11):e183 [FREE Full text] [CrossRef] [Medline]
  60. Lelutiu-Weinberger C, Pachankis JE, Gamarel KE, Surace A, Golub SA, Parsons JT. Feasibility, acceptability, and preliminary efficacy of a live-chat social media intervention to reduce HIV risk among young men who have sex with men. AIDS Behav 2015 Jul;19(7):1214-1227 [FREE Full text] [CrossRef] [Medline]
  61. Mi G, Wu Z, Wang X, Shi CX, Yu F, Li T, et al. Effects of a quasi-randomized web-based intervention on risk behaviors and treatment seeking among HIV-positive men who have sex with men in Chengdu, China. Curr HIV Res 2015;13(6):490-496 [FREE Full text] [CrossRef] [Medline]
  62. Mikolajczak J, Breukelen van G, Kok G, Hospers H. Evaluation of an online HIV-prevention intervention to promote HIV-testing among men who have sex with men: a randomised controlled trial. Neth J Psychol 2012;67(2):21-35 [FREE Full text]
  63. Mimiaga MJ, Thomas B, Biello K, Johnson BE, Swaminathan S, Navakodi P, et al. A pilot randomized controlled trial of an integrated in-person and mobile phone delivered counseling and text messaging intervention to reduce HIV transmission risk among male sex workers in Chennai, India. AIDS Behav 2017 Nov;21(11):3172-3181 [FREE Full text] [CrossRef] [Medline]
  64. Mustanski B, Garofalo R, Monahan C, Gratzer B, Andrews R. Feasibility, acceptability, and preliminary efficacy of an online HIV prevention program for diverse young men who have sex with men: the keep it up! intervention. AIDS Behav 2013 Nov;17(9):2999-3012 [FREE Full text] [CrossRef] [Medline]
  65. Mustanski B, Parsons JT, Sullivan PS, Madkins K, Rosenberg E, Swann G. Biomedical and Behavioral Outcomes of Keep It Up!: An eHealth HIV Prevention Program RCT. Am J Prev Med 2018 Aug;55(2):151-158 [FREE Full text] [CrossRef] [Medline]
  66. Nöstlinger C, Platteau T, Bogner J, Buyze J, Dec-Pietrowska J, Dias S, Eurosupport Study Group. Implementation and operational research: computer-assisted intervention for safer sex in HIV-positive men having sex with men: findings of a European randomized multi-center trial. J Acquir Immune Defic Syndr 2016 Mar 1;71(3):e63-e72 [FREE Full text] [CrossRef] [Medline]
  67. Patel VV, Rawat S, Lelutiu-Weinberger C, Dange A, Kamath C, Poojary R, et al. CHALO! A social media based peer-delivered intervention increases HIV testing in men who have sex with men in Mumbai, India: a randomized trial. In: Oral abstracts of the 21st International AIDS Conference.: Journal of the International AIDS Society; 2016 Presented at: the 21st International AIDS Conference; 18–22 July 2016; Durban p. 103   URL:
  68. Prati G, Mazzoni D, Cicognani E, Albanesi C, Zani B. Evaluating the persuasiveness of an HIV mass communication campaign using gain-framed messages and aimed at creating a superordinate identity. Health Commun 2016 Sep;31(9):1097-1104. [CrossRef] [Medline]
  69. Read SJ, Miller LC, Appleby PR, Nwosu ME, Reynaldo S, Lauren A, et al. Socially optimized learning in a virtual environment: Reducing risky sexual behavior among men who have sex with men. Human Comm Res 2006 Jan;32(1):1-34. [CrossRef]
  70. Reback CJ, Fletcher JB, Swendeman DA, Metzner M. Theory-based text-messaging to reduce methamphetamine use and HIV sexual risk behaviors among men who have sex with men: automated unidirectional delivery outperforms bidirectional peer interactive delivery. AIDS Behav 2019 Jan;23(1):37-47 [FREE Full text] [CrossRef] [Medline]
  71. Reback CJ, Grant DL, Fletcher JB, Branson CM, Shoptaw S, Bowers JR, et al. Text messaging reduces HIV risk behaviors among methamphetamine-using men who have sex with men. AIDS Behav 2012 Oct;16(7):1993-2002 [FREE Full text] [CrossRef] [Medline]
  72. Rhodes SD, McCoy TP, Tanner AE, Stowers J, Bachmann LH, Nguyen AL, et al. Using social media to increase HIV testing among gay and bisexual men, other men who have sex with men, and transgender persons: Outcomes from a randomized community trial. Clin Infect Dis 2016 Jun 1;62(11):1450-1453 [FREE Full text] [CrossRef] [Medline]
  73. Rhodes SD, Vissman AT, Stowers J, Miller C, McCoy TP, Hergenrather KC, et al. A CBPR partnership increases HIV testing among men who have sex with men (MSM): outcome findings from a pilot test of the CyBER/testing internet intervention. Health Educ Behav 2011 Jun;38(3):311-320 [FREE Full text] [CrossRef] [Medline]
  74. Rosser BR, Oakes JM, Konstan J, Hooper S, Horvath KJ, Danilenko GP, et al. Reducing HIV risk behavior of men who have sex with men through persuasive computing: results of the Men's INTernet Study-II. AIDS 2010 Aug 24;24(13):2099-2107 [FREE Full text] [CrossRef] [Medline]
  75. Schonnesson LN, Bowen AM, Williams ML. Project SMART: Preliminary results from a test of the efficacy of a Swedish internet-based HIV risk-reduction intervention for men who have sex with men. Arch Sex Behav 2016 Aug;45(6):1501-1511. [CrossRef] [Medline]
  76. Solorio R, Norton-Shelpuk P, Forehand M, Montaño D, Stern J, Aguirre J, et al. Tu Amigo Pepe: Evaluation of a multi-media marketing campaign that targets young Latino immigrant MSM with HIV testing messages. AIDS Behav 2016 Sep;20(9):1973-1988. [CrossRef] [Medline]
  77. Tang W, Wei C, Cao B, Wu D, Li KT, Lu H, et al. Crowdsourcing to expand HIV testing among men who have sex with men in China: a closed cohort stepped wedge cluster randomized controlled trial. PLoS Med 2018 Aug;15(8):e1002645 [FREE Full text] [CrossRef] [Medline]
  78. Uhrig JD, Lewis MA, Bann CM, Harris JL, Furberg RD, Coomes CM, et al. Addressing HIV knowledge, risk reduction, social support, and patient involvement using SMS: results of a proof-of-concept study. J Health Commun 2012;17(Suppl 1):128-145. [CrossRef] [Medline]
  79. Wang Z, Lau JT, Ip M, Ho SP, Mo PK, Latkin C, et al. A randomized controlled trial evaluating efficacy of promoting a home-based HIV self-testing with online counseling on increasing HIV testing among men who have sex with men. AIDS Behav 2018 Jan;22(1):190-201. [CrossRef] [Medline]
  80. Ybarra ML, Prescott TL, Phillips GL, Bull SS, Parsons JT, Mustanski B. Pilot RCT results of an mHealth HIV prevention program for sexual minority male adolescents. Pediatrics 2017 Jul;140(1):- [FREE Full text] [CrossRef] [Medline]
  81. Young SD, Cumberland WG, Nianogo R, Menacho LA, Galea JT, Coates T. The HOPE social media intervention for global HIV prevention in Peru: a cluster randomised controlled trial. Lancet HIV 2015 Jan;2(1):e27-e32 [FREE Full text] [CrossRef] [Medline]
  82. Zou H, Fairley CK, Guy R, Bilardi J, Bradshaw CS, Garland SM, et al. Automated, computer generated reminders and increased detection of gonorrhoea, chlamydia and syphilis in men who have sex with men. PLoS One 2013;8(4):e61972 [FREE Full text] [CrossRef] [Medline]
  83. Flores SA, Crepaz N, HIV Prevention Research Synthesis Team. Quality of study methods in individual- and group-level HIV intervention research: critical reporting elements. AIDS Educ Prev 2004 Aug;16(4):341-352. [CrossRef] [Medline]
  84. Higgins JP, Deeks JJ, Altman DG. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0.: The Cochrane Collaboration; 2011. 16.5 Studies With More Than Two Intervention Groups   URL: [accessed 2018-09-01]
  85. Portnoy DB, Scott-Sheldon LA, Johnson BT, Carey MP. Computer-delivered interventions for health promotion and behavioral risk reduction: a meta-analysis of 75 randomized controlled trials, 1988-2007. Prev Med 2008 Jul;47(1):3-16 [FREE Full text] [CrossRef] [Medline]
  86. Cushing CC, Steele RG. A meta-analytic review of eHealth interventions for pediatric health promoting and maintaining behaviors. J Pediatr Psychol 2010 Oct;35(9):937-949. [CrossRef] [Medline]
  87. Higgins JP, Deek JJ. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0.: The Cochrane Collaboration; 2011. Post-Intervention Versus Change From Baseline   URL: https:/​/handbook-5-1.​​chapter_7/​7_7_3_1_post_intervention_versus_change_from_baseline.​htm [accessed 2018-09-01]
  88. Deeks JJ, Higgins JP, Altman DG. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0.: The Cochrane Collaboration; 2011. The Standardized Mean Difference   URL: [accessed 2018-09-01]
  89. Morris SB, DeShon RP. Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychol Methods 2002 Mar;7(1):105-125. [CrossRef] [Medline]
  90. Wilson DB. The Campbell Collaboration. Fairfax: George Mason University; 2001. Practical Meta-Analysis Effect Size Calculator   URL: [accessed 2018-09-01]
  91. Lenhard W, Lenhard A. Psychometrica. Dettelbach: Psychometrica; 2016. Computation of Effect Sizes   URL: [accessed 2018-09-01]
  92. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Abingdon, United Kingdom: Routledge; 2013.
  93. Deeks JJ, Higgins JP, Altman DG. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0.: The Cochrane Collaboration; 2011. 9.5.2 Identifying and Measuring Heterogeneity   URL: [accessed 2018-09-01]
  94. Viechtbauer W, Cheung MW. Outlier and influence diagnostics for meta-analysis. Res Synth Methods 2010 Apr;1(2):112-125. [CrossRef] [Medline]
  95. Sterne JA, Egger M, Moher D. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0.: The Cochrane Collaboration; 2011. Recommendations on Testing for Funnel Plot Asymmetry   URL: https:/​/handbook-5-1.​​chapter_10/​10_4_3_1_recommendations_on_testing_for_funnel_plot_asymmetry.​htm [accessed 2018-09-01]
  96. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0.: The Cochrane Collaboration; 2011. Ensure That There Are Adequate Studies to Justify Subgroup Analyses and Meta-Regressions   URL: https:/​/handbook-5-1.​​chapter_9/​9_6_5_1_ensure_that_there_are_adequate_studies_to_justify.​htm
  97. Viechtbauer W. Conducting Meta-Analyses in R with the metafor Package. J Stat Softw 2010;36(3):1-48. [CrossRef]
  98. Smith DK, Herbst JH, Zhang X, Rose CE. Condom effectiveness for HIV prevention by consistency of use among men who have sex with men in the United States. J Acquir Immune Defic Syndr 2015 Mar 1;68(3):337-344. [CrossRef] [Medline]
  99. Garnett C, Crane D, Brown J, Kaner E, Beyer F, Muirhead C, et al. Reported theory use by digital interventions for hazardous and harmful alcohol consumption, and association with effectiveness: meta-regression. J Med Internet Res 2018 Feb 28;20(2):e69 [FREE Full text] [CrossRef] [Medline]
  100. Danaher BG, McKay HG, Seeley JR. The information architecture of behavior change websites. J Med Internet Res 2005 May 18;7(2):e12 [FREE Full text] [CrossRef] [Medline]
  101. Lehto T, Oinas-Kukkonen H. Persuasive features in web-based alcohol and smoking interventions: a systematic review of the literature. J Med Internet Res 2011 Jul 22;13(3):e46 [FREE Full text] [CrossRef] [Medline]
  102. Reback CJ, Ling D, Shoptaw S, Rohde J. Developing a text messaging risk reduction intervention for methamphetamine-using MSM: research note. Open AIDS J 2010 May 14;4:116-122 [FREE Full text] [CrossRef] [Medline]
  103. Tso LS, Tang W, Li H, Yan HY, Tucker JD. Social media interventions to prevent HIV: a review of interventions and methodological considerations. Curr Opin Psychol 2016 Jun 1;9:6-10 [FREE Full text] [CrossRef] [Medline]
  104. Spinner CD, Boesecke C, Zink A, Jessen H, Stellbrink H, Rockstroh JK, et al. HIV pre-exposure prophylaxis (PrEP): a review of current knowledge of oral systemic HIV PrEP in humans. Infection 2016 Apr;44(2):151-158. [CrossRef] [Medline]
  105. Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, et al. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int J Technol Assess Health Care 2012 Apr;28(2):138-144. [CrossRef] [Medline]

CMC: computer-mediated communication
eHealth: electronic health
HCI: human-computer interaction
IMB: information-motivation-behavioral skills model
MSM: men who have sex with men
MSP: multiple sex partnership
STI: sexually transmitted infection
UAI: unprotected anal intercourse

Edited by G Eysenbach; submitted 26.08.19; peer-reviewed by F Yu, Z Wang; comments to author 28.09.19; revised version received 19.12.19; accepted 01.02.20; published 25.05.20


©Meiqi Xin, Kasisomayajula Viswanath, Angela Yuen-Chun Li, Wangnan CAO, Yuhong HU, Joseph Tak-Fai Lau, Phoenix Kit-Han Mo. Originally published in the Journal of Medical Internet Research (, 25.05.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.