Treatment content is characterized by the use of theory for intervention development, as theoretical constructs can be operationalized into treatment techniques [21]. Significantly greater reductions in unprotected anal intercourse (UAI) among MSM have been observed for interventions reporting any use of theory [9], yet it is unclear whether the extent of use predicts intervention effectiveness, particularly in the eHealth context. A meta-analysis of internet-based behavioral interventions across health domains revealed a significant association between more extensive theory use and increased effect sizes [22].

Treatment content also features the application of tailoring strategies, defined as the integration of recipients’ responses into the intervention system to generate user-driven content [23]. There are 3 tailoring types: (1) feedback: providing unique recommendations derived from an assessment of individual needs or characteristics related to a given behavior; (2) adaptation: matching content to a relevant group based on known behavioral determinants; and (3) personalization: customizing content with personally identifiable information [24]. Computer technology–based interventions with individually tailored content adapted to the stage of change were shown to be more efficacious in increasing condom use in the general population [14].

The programming features of a treatment can strongly determine the intervention intensity. The amount of accessible content per interaction with the intervention differs by navigation style: tunneled interventions deliver treatment through a predetermined sequence of steps, whereas self-paced interventions release content all at once and permit recipients to control the navigation [25-27]. Existing evidence is inconclusive regarding the influence of navigation style on the effectiveness of eHealth interventions [26,28,29]. Treatment duration, referring to the time span of delivery, represents the overall intervention burden. Larger effects and higher adherence have been generally reported for online behavioral interventions of a shorter duration [23].

On the basis of the nature of users’ interaction with eHealth platforms, 2 operation modes of technology use for intervention delivery have been distinguished: (1) human-computer interaction (HCI), featuring direct interaction with a computerized system and automated delivery of a preprogrammed treatment and (2) computer-mediated communication (CMC), featuring remote delivery through interpersonal communications via eHealth media. These are considered to be inseparable, albeit essentially different aspects of eHealth technology and capable of deploying similar treatment strategies [25,30]. It has been reported that the provision of remote therapeutic support, not fully automated treatment, achieved significantly better psychological outcomes than passive control [31].

Four modality types have been applied to eHealth interventions for HIV prevention, and each of them possesses a unique capacity to facilitate intervention delivery [11,19,32]. The 2 most common modalities are the interactive module, in which users actively engage in an intervention following a preset workflow, and the static site, in which users passively receive prescriptive information [30]. A previous review suggested the superiority of interactive over static interventions in predicting increased condom use and sexually transmitted infection (STI) testing [12]. The strength of text messaging lies in the ubiquitous use of mobile devices to deliver real-time personalized interventions [11,33]. It can incorporate effective communication techniques and has demonstrated its efficacy in promoting health behavior change [33,34]. This study uses the broader term, “messaging,” to encompass related technologies that enable multimedia delivery [33]. Social media has emerged as a novel modality that features the creation and sharing of user-generated content in an online community [32,35], which can effectively promote HIV testing [35].

Intervention effectiveness observed in real-world practice can change with actual exposure to efficacious intervention components, and nonusage attrition is common in eHealth interventions [36]. Intervention adherence is conceptualized as the proportion of participants who engage in the intervention as prescribed to achieve a desired effect [29]. This allows for a comparable measurement of intervention exposure across use parameters, by contrasting the actual usage of an intervention during its implementation with the intended usage predefined at the design stage [25]. Adherence has been found to significantly predict the effectiveness of eHealth interventions [23].

This study aimed to conduct a meta-analysis of the effectiveness of eHealth technology–based interventions for promoting HIV-preventive behaviors among MSM and to provide an in-depth investigation of effectiveness predictors.

Data were extracted from each eligible program and its supplementary materials by one coder (MX) and double-checked by a second coder (AL). Information was first gathered on factors related to the program design, which were used to define the coding unit and to determine the grouping sets for the primary meta-analysis.

One intervention condition and one comparator were selected to form an independent study (a pairwise comparison) for the meta-analysis. If more than one intervention group was reported, each of them was separately included, given the potential variation in interventions, while the initial control group was proportionally divided up among intervention conditions to ensure independence [84]. If more than one control group was reported, the less intensive one was included for ease of interpretation [85]. If an eligible concurrent comparator was not obtained, or independent group contrasts were not the program focus, the preintervention status was treated as the comparator [86]. Finally, for each established comparison, identification items (eg, author) and sample characteristics (eg, age) were coded.

Study design was classified according to the outcome assessment points and comparison status (ie, using a concurrent or preintervention group). Comparator type was coded as (1) passive, including blank, waitlist, and attention controls or (2) active, including non-eHealth and lower tech eHealth treatment. Follow-up period was classified according to the interval between the intervention end and the follow-up points: (1) immediately postintervention; (2) short-term: ≤3 months; or (3) long-term follow-up: >3 months. The outcome measures with adequate studies for effect size pooling were UAI, condom use, HIV testing, and multiple sex partnership (MSP).

A cluster of prespecified factors that captured the intervention characteristics were coded within the conceptualized framework described earlier. A relative coding rule was applied, that is, where a certain item was identical across the paired conditions, the study was coded as not having that feature, as it could not explain the conditional difference in the effectiveness.

With regard to the key features of treatment content, the extent of theory use was first assessed using the modified Theory Coding Scheme [21]. A total of 11 items pertinent to intervention development were used to compute an overall score [22], with a higher value indicating more extensive use of the theory. The second content feature was the use of tailoring strategies, which was coded in both ways, binary (whether adopting the feedback, adaptation, or personalization strategy) and continuous (the number of strategies adopted). The programming-specific treatment features were categorized into mutually exclusive groups: navigation style (self-paced or tunneled) and treatment duration (single session, ≤1 month, >1-3 months, and >3 months).

The utilization of eHealth technology was characterized by the operation mode (ie, HCI and CMC) and the modality type (ie, static site, interactive module, messaging, and social media). For both variables, interventions with the presence of more than one feature were grouped into a combined category to allow for exclusive coding. As indicators of intervention exposure, intended usage was first extracted and varying levels of actual usage were identified accordingly. Intervention adherence was then calculated as the percentage of participants whose actual usage matched the intended usage [25].

Standardized mean differences between conditions (Cohen d) and their standard error at each follow-up point were derived to represent the magnitude of intervention effects [87,88]. Following Morris and DeShon’s [89] procedures, the effect size estimate and its sampling error were computed or transformed to be scaled on a common “raw-score” metric, creating a synthesis across the study design. Unadjusted outcome measurements were retrieved to establish comparability across estimates. For studies in which means and standard deviations were not provided, effect measures reported in other forms (eg, risk ratio) were converted to the Cohen d statistic using well-developed calculators [90,91].

As recommended, UAI and condom use were combined into one outcome type, “UAI” [14]. For studies that used multiple instruments to measure an outcome, the most common instrument was chosen (eg, the frequency of UAI was prioritized over the count of UAI partners [56]). For studies measuring different subtypes of a certain outcome on the same scale (eg, serostatus-specific UAI [71]) or reporting subgroup effects by significant moderators that were unrelated to intervention components (eg, affect level [44]), all effect sizes were aggregated meta-analytically within the study. Consequently, each study yielded only one Cohen d value per outcome.

With an assumption of the intervention diversity, pooled effect sizes (d₊) and 95% confidence intervals were generated using random effects models with the inverse variance weighting method. Values such as 0.2, 0.5, and 0.8 were interpreted as small, moderate, and large, respectively [92]. Primary meta-analyses were performed to combine effect sizes within different study sets for each outcome: (1) an overall estimate at the end point (one effect arising per study); (2) group-specific estimates of the end point effect by comparison status (nested groupings); and (3) group-specific estimates by follow-up period (a multi-wave study contributing multiple effects). To maximize analytical power, the first type of estimation based on a full data set was used for further analyses.

Heterogeneity across studies was assessed by using the Q statistic with a P value <.05, indicating the presence of significant heterogeneity, and quantified by using the I² statistic, where a value of 30% to 60% denoted “moderate” and ≥75% denoted “considerable” heterogeneity [93]. Wherever considerable heterogeneity was found, outliers were identified as studies that significantly distorted the pooled effect using influence analyses [94] and were removed to ensure the accuracy and generalizability of the findings. Publication bias was then assessed by visually inspecting the funnel plot of effect sizes and conducting an Egger regression test to examine the plot asymmetry when there were at least 10 estimates [95].

Next, secondary analyses were performed to test the moderation effect of intervention features where adequate studies were available (n≥10) [96]. Subgroup and meta-regression analyses were applied for dichotomous and continuous measures, respectively; a significant moderator was indicated by a P value <.05 for the heterogeneity across subgroups (Q_b) or the regression coefficient (β). All analyses were conducted in R 3.5.2 with the metafor package (Wolfgang Viechtbauer) [97].

In combing all 35 studies that tested for UAI, a significant and small overall intervention effect was observed at end point (d₊=−.32; P<.001), albeit with considerable heterogeneity (I²=83.3%). One study included subjects who reported recent condom-less sex with partners of either gender before enrollment; such a high-risk preintervention status might have contributed to the greatly decreased UAI (d=−1.13; SE 0.21) [49]. Moreover, the sample had the highest mean age of 45.1 years among all studies. Another study was focused on MSM sex workers and primarily addressed the context of transactional sex [63]. An extremely positive effect was found for reduced UAI with nonpaying male partners (d=−3.95; SE 0.35). Removing both outliers resulted in a smaller effect with moderate heterogeneity (d₊=−.21; P<.001; I²=50.8%).

When stratified by the partner-specific outcome, a larger end point d₊ was shown for UAI with main partners than with nonmain partners. When stratified by the follow-up period, a significant d₊ was obtained at all points. The nested subgroup analysis found a significantly greater intervention effect when compared with preintervention groups than with concurrent groups (Q_b=13.38; P<.001); hence, the meta-analysis was performed separately for the 2 types of comparison status as recommended (Figures 2 and 3) [89].

Similarly, pooling the studies that tested for HIV testing revealed a significant, small overall effect at endpoint (n=23; d₊=−.32; P<.001). The considerable heterogeneity (I²=84.0%) identified was largely attributable to the detection of outliers. Bourne et al [40] tested a texting reminder intervention against a blank control group involving participants who had refused such reminders; thus, the extreme effect size (d=.80; SE 0.06) could have been susceptible to volunteer bias. Wang et al [79] promoted a novel self-testing approach and distributed free home-based testing kits; the uptake of any type of testing was found to be considerably higher in the intervention group (d=1.18; SE 0.14). Mikolajczak et al [62] unexpectedly found a negative but nonsignificant intervention effect on HIV or STI testing relative to an active comparator (d=−.13; SE 0.12). The exclusion of outliers caused a slight change in the effectiveness but substantially reduced heterogeneity (d₊=.38; P<.001; I²=64.2%; Figure 4). A larger d₊ was further shown at postintervention than at short-term follow-up; however, only one sample was followed up for more than 3 months. No significant group difference by comparison status was observed.

A significant, small overall effect was observed on the reduced MSP at endpoint with moderate heterogeneity (d₊=−.26; P=.02; n=6; I²=60.2%; Figure 5). The largest d₊ was shown at medium-term follow-up among all follow-up periods, although only a few studies were available for each grouping. No between-group test was performed for comparison status because of the small number of studies. Publication bias was detected for none of the outcomes. More results of primary analyses are presented in Table 2.

Most of the studies were designed based on a theory (n=37); the most common one was the information-motivation-behavioral skills model (IMB; n=14). The intervention effects on UAI and HIV testing appeared to be comparable regardless of whether any theory was used. Among the theory-based studies, IMB was significantly associated with greater UAI reductions than other theories for concurrent comparisons (Q_b=4.33; P=.04). The more extensive use of tailoring strategies was significantly associated with decreased UAI for concurrent comparisons (β=−.08; P=.01). Specifically, feedback was the only strategy that had a significantly negative effect on UAI (Q_b=4.00; P=.04), whereas adopting the personalization strategy tended to have a positive effect on HIV testing (Q_b=2.76; P=.10).

Significantly greater UAI reductions were shown for the tunneled (Q_b=7.23; P=.01) and self-paced (Q_b=4.23; P=.04) treatments relative to concurrent and preintervention groups, respectively. Such a moderating effect of navigation style was not detected for HIV testing. Longer treatments showed a significantly greater increase in HIV testing than those with a single session or those lasting ≤1 month (Q_b=16.97; P<.001). However, the effects on UAI did not significantly differ across treatment durations.

Over half of the interventions were operated via HCI (n=26), and the others were in a CMC (n=10) or combined (n=8) mode. More studies used a single modality (n=32) than those incorporating multi-modalities in the same intervention albeit with varying combination patterns (n=14). No between-group difference was observed in the intervention effects on UAI by operation mode or modality type. The CMC and combined modes tended to present a larger d₊ value for HIV testing than did the HCI mode (Q_b=2.76; P=.09). When separately exploring the effect of specific modes, the use of CMC predicted a significantly greater increase in HIV testing (Q_b=4.38; P=.04), whereas HCI use showed no association with effectiveness. Moreover, a significantly larger d₊ value was found for the use of messaging, social media, and multi-modalities than for static sites (Q_b=12.79; P=.005).

Among all interventions, the adherence rates ranged from 25.5% to 100%. A higher adherence rate was significantly associated with a decrease in UAI for concurrent comparisons (β=−.27; P=.03). Given the highly negative skewed distribution, the studies were evenly divided into 3 levels to reflect the degree of relative adherence; the adherence rates were above 91.6% in the first tertile and below 76% in the third tertile. Intervention effects on HIV testing significantly differed by adherence level (Q_b=7.28; P=.03). A moderate d₊ value was observed for high-level studies (in the first tertile) and a small one for mid- and low-level studies. Details of the intervention features are presented in Multimedia Appendix 4; more results of moderation analyses are presented in Multimedia Appendix 5.

This study confirmed the previous finding [14] that the use of the theory did not moderate the effectiveness of eHealth-based interventions for HIV prevention, although the majority of treatments were developed on a theoretical basis. The overall extent of theory use failed to significantly influence effectiveness across outcomes. However, this finding may merely reflect a lack of sensitivity in distinguishing an effect [99]. The number of tailoring strategies used significantly moderated effectiveness in reducing UAI, and only the most frequently used strategy, that is, feedback, was further shown to be effective. Treatments were less tailored for interventions promoting HIV testing, and only the moderating effect of personalization use reached marginal significance. Some evidence supporting the superiority of certain strategies has been reported [23]. The use of feedback and personalization commonly featured tailoring at an individual level, whereas the use of adaptation generated content matched to group-level factors.

Inconclusive results were found for the role of navigation style. Tunneled treatments achieved greater UAI reductions than self-paced treatments in studies using a concurrent group but lower reductions in those using a preintervention group. A plausible explanation is that the tunneling pattern differed across comparison types. A tunneled treatment by definition requires multiple interactions with the intervention, which could impair the effectiveness of eHealth behavioral interventions [23]. Over 60% of treatments with a preintervention group involved more than 5 modules, whereas only 24% of those with a concurrent group did so. Above all, the efficacy of the tunneling design itself is controversial as, on the one hand, the sequential release could ameliorate information anxiety and enhance the behavioral change process, whereas, on the other hand, the artificial confines could inhibit typical information-foraging behavior and intervention participants may lack the motivation to accommodate such constraints [100,101].

Longer treatments significantly predicted higher uptake of HIV testing but not reduced UAI, which adds to the evidence in favor of a difference in the influence of treatment duration on sexual and detection behaviors [12]. However, this finding might merely reject a linear relationship between treatment duration and the effect on UAI. A recent meta-analysis even demonstrated a negative effect of increased intervention length on intervention adherence and behavioral impacts, driven by decreased motivation over time [23].

Taking advantage of both human communication (eg, flexibility and rapport building [49]) and eHealth-enabled capacities (eg, convenience and anonymity [30]), the use of CMC was shown to be effective in increasing HIV testing. There was little evidence for the moderating effect of the operation mode on UAI, although only 1 intervention used a CMC mode. The results illustrate the need to incorporate human involvement into eHealth interventions. A variety of communicative functions have proved their efficacy in promoting behavior change, including counselor- or user-initiated conversations and peer-to-peer interactions [22]. Moreover, the greatest effectiveness was obtained with a combined mode, so the relevant question is not whether HCI or CMC is superior but, rather, how to combine them to maximize persuasiveness.

Similarly, intervention effects on the outcome of HIV testing but not UAI varied by modality type. Messaging, social media, and the combined type achieved a comparably small-to-medium effect. Messaging, largely operated via portable devices, affords individualized interventions (eg, location-based services) and dynamic connections (eg, response on request) [11,102]. Social media stands out for its capabilities to influence social norms and create a peer-supportive environment [103], especially within a closely connected community such as MSM [35]. It also enables interventions to blend into users’ daily lives by utilizing well-established platforms (eg, Facebook) [32]. The use of hybrid modalities may further generate a synergistic effect.

Consistent with previous literature [23], a high degree of adherence to eHealth interventions predicted a protective effect on UAI and HIV testing. Assuming intention-to-treat as the motive to engage in an intervention, adherence is rooted in the properties of intervention design. Efficacious strategies (eg, self-monitoring), observable benefits (eg, health outcomes), and feasible programming (eg, appropriate workload) can all bolster engagement [26,36]. Usability of eHealth technology is also critical to minimizing discontinuance [36].