Data were collected from January 24, 2018, to October 31, 2019. The methods for the study have been previously described [28] and are summarized here. Eligibility criteria are summarized in Table 1. Men were classified as (1) HIV seropositive, (2) HIV seronegative at high risk (any anal sex not protected by condoms or not taking PrEP, as prescribed, in the past 3 months), and (3) HIV seronegative at low risk (anal sex protected by condoms in the past 3 months or taking PrEP, as prescribed, in the past 3 months, ie, adherent PrEP users). Owning a cell phone was also required, and smartphone literacy was thus a de facto eligibility criterion. This research was reviewed and approved by the Emory University institutional review board (protocol IRB00087684) on June 13, 2016. We collected individual informed consent in person at the study enrollment visit. Consenting participants were randomly assigned to either the immediate intervention group or the waitlist control group in a 1:1 ratio within 36 strata based on the 3 serostatus/risk groups, 3 cities, race/ethnicity (non-Hispanic White vs other race/ethnicity), and phone type (Android vs iOS) [31]. Participants in the intervention group received one 2-sentence message every other day and a 1-minute video weekly, in addition to access to HIV and STI kit ordering and informational resources.

At in-person baseline visits, participants assigned to the intervention group downloaded and received orientation to the study app and received access to the mobile app and prevention messages based on their risk profiles. Participant contact and retention activities were supported through the Emory Study Management and Retention Tool (SMART). The intervention was provided over 3 months, after which the app was deactivated. Waitlist control participants were not provided access to the intervention app at the baseline visit but continued to receive quarterly surveys. An overview of the study timeline and complete survey instruments have been published and are available online [28].

There were multiple prevention outcomes, which varied among the 3 risk groups, as previously reported [28]. The specific outcomes and measures used are summarized in Table 2.

Analyses were stratified by the baseline HIV status/risk group: high-risk HIV-negative, low-risk HIV-negative, and living with HIV (LWH) groups. Counts and relative frequencies were used to describe sociodemographics and sexual history at baseline by intervention status (intervention group vs control group) and HIV status/risk group, and chi-square tests assessed for differences in these characteristics by intervention status (Table 3). To account for within-person repeated measures and randomization strata, separate mixed-effect logistic models assessed for association over time of the intervention with each outcome.

A post hoc analysis was conducted to compare control participants with intervention participants who had at least 30 minutes of intervention use in order to assess the potentially mediating effect of the time spent on the app; 30 minutes was the estimated time required to read and rate all written messages and view the video messages. A second post hoc analysis was conducted because not all participants who ordered and received at-home STI test kits returned them. In this second analysis, we defined the intention to test for STIs as either ordering an STI test kit through the mobile app or reporting having taken an STI test. Result reporting was prepared in accordance with Consolidated Standards of Reporting Trials (CONSORT) guidelines [33].

From January 24, 2018, to November 2, 2018, a total of 9966 GBMSM were assessed for eligibility for the randomized controlled trial; reasons for lack of eligibility are shown in Figure 1. Specifically, 2841 (28.51%) of 9966 GBMSM screened were eligible and provided contact information, of which 1236 (43.51%) attended a baseline study visit, 1226 (99.19%) consented and were randomized to intervention (n=611, 49.84%) and control (n=615, 50.16%) arms (Figure 1).

The analytic sample comprised 1220 GBMSM enrolled in the study, including 608 (49.84%) randomized to the intervention arm and 612 (50.16%) randomized to the control arm (Figure 2). There was no failure of randomization within the 3 risk groups, except for unprotected anal intercourse among LWH GBMSM (Table 2). Most participants were GBMSM of color (n=378 [30.98%] Black/African American, n=183 [15%] Hispanic/Latino, n=148 [12.13%] other/mixed), were aged ≥30 years (772/1220 [63.28%]), and had at least a 4-year college education (654/1220 [53.61%]). Other indicators of risk and prevention services did not differ between control and intervention arms (data not shown). Self-reported levels of use of prevention and care services were often less than recommended by the Centers for Disease Control and Prevention (CDC). High-risk HIV-negative GBMSM reported moderate levels of STI (165/427, 38.6%) and HIV (162/427, 37.9%) testing in the 3 months before baseline; only 22 of 427 (5.1%) reported using PrEP, and adherence to PrEP was low (11/22, 50%, reported ≥25/30 pills in the prior 30 days). A high proportion (392/427, 91.8%) reported sex not protected by PrEP or condoms in the past 3 months. Among low-risk HIV-negative GBMSM, baseline use of STI (254/410, 61.9%) and HIV (259/410, 63.2%) testing was higher; in addition, 222 of 410 (54.1%) reported being on PrEP, with high (212/222, 95.5%) adherence. Many HIV-negative GBMSM were designated as low risk because they were on PrEP; routine PrEP care includes HIV and STI testing. Among LWH GBMSM, most (271/383, 70.8%) reported recent STI testing, being on antiretroviral therapy (ART; 368/383, 96.1%) and high ART adherence (335/368, 91%). About 1 in 4 reported anal sex not protected by condoms in the past 3 months (98/383, 25.6%), and most (251/383, 65.5%) reported engaging in HIV care during the prior quarter.

Retention was 1065 of 1226 (86.87%) at 3 months, 1046 of 1226 (85.32%) at 6 months, and 985 of 1226 (80.34%) at 9 months. Factors associated with retention were consistent across time points: retention was higher among participants who were White non-Hispanic, had more education, were employed full-time, and were not homeless. At the 9-month follow-up, higher retention was also associated with gay/homosexual identity (vs bisexual or other identities) and online recruitment (vs venue-based recruitment).

Intervention efficacy was analyzed by HIV status/risk group (Table 4). Among high-risk HIV-negative GBMSM, the odds of HIV testing (Figure 3) were higher at the immediate postintervention time point and the odds of current PrEP use (Figure 4) were higher at the 3-month postintervention time point, but not at 6 or 9 months postintervention (Table 4). Of the high-risk GBMSM in the intervention arm who tested for HIV in the immediate postintervention period (n=126), 16 (12.7%) used an at-home HIV self-test kit provided through the app. As shown in Figure 4, although the initial prevalence of PrEP use was nonsignificantly lower in the intervention group at baseline, PrEP use rose from 3% at baseline to 15% by the 3-month follow-up assessment and remained at 15% through the 6-month follow-up assessment. Although odds ratio (OR) estimates comparing intervention to control for PrEP adherence were >1.0 at all postintervention time points for high-risk GBMSM, they were not statistically significant, given the small number of high-risk HIV-negative GBMSM taking PrEP.

For low-risk HIV-negative GBMSM, there were no changes in study outcomes associated with assignment to the intervention. For LWH GBMSM, STI testing was significantly lower in the immediate posttest period in the intervention group compared to the control group.

Given the potential importance of race/ethnicity, age group, and education level as potential moderators of intervention effects, we conducted stratified analyses of the outcome variables for each of the 3 HIV status/risk groups for the following subgroups: non-Hispanic White versus others; age 18-29 versus ≥30 years; and <4-year college degree versus 4-year degree or more. No meaningful difference in outcomes was identified among stratum-specific estimates. Further, a sensitivity analysis assessed the time spent on the mobile app over the 3-month intervention (control vs ≥30 minutes of intervention use) for the 3 HIV status/risk groups. No meaningful or significant difference in outcomes was identified.

In a posthoc analysis, the intention to test for STIs (ie, ordering an STI test kit or reporting having had an STI test) was higher in all HIV-negative participants (low-risk HIV-negative: aOR 3.1, 95% CI 1.7-5.6; high-risk HIV-negative: aOR 3.5, 95% CI 2.0-6.2; all HIV-negative regardless of risk group: aOR 3.4, 95% CI 2.3-5.1).

We tested an app-based intervention to provide messages to promote HIV prevention and care and provide access to core HIV and STI prevention services to GBMSM. The results showed significant increases in HIV screening during the period of app use among high-risk HIV-negative GBMSM. The results also indicated increased prevalence of PrEP use among high-risk HIV-negative GBMSM in the 3-month period after app use. We identified no protective changes in study outcomes for low-risk HIV-negative GBMSM or LWH GBMSM. Given the high baseline PrEP use and adherence among low-risk HIV-negative GBMSM who did not report recent anal sex not protected by condoms and high levels of ART use and the high baseline adherence among LWH GBMSM, there was little room for improvement statistically among these groups on key outcome variables. We recognize, however, that the vulnerability of GBMSM may change over time (ie, seasons of risk [34]), so the utility of the app and its messaging might change over time for individual men.

We developed our intervention in light of 3 principles. First, we worked from the paradigm of integrated HIV prevention and care continua. Although the HIV status was determined at the beginning of the trial, when the app is used in practice, HIV testing will be the starting point for both continua. Thus, the intervention could serve GBMSM who do not know their HIV status [35]. Second, we sought to develop an intervention that could be offered to all GBMSM rather than to demographic or risk subsets of men. Third, we sought to develop an intervention that could be scaled to provide intervention content to large numbers of men without the need for a large staff of interventionists. A free, downloadable app that disseminates one 2-sentence message every other day and a 1-minute video weekly might be easily incorporated into a user’s life without requiring the time and travel often required of more traditional, in-person interventions. Further, mobile app interventions are a potentially low-cost strategy from the perspective of health departments and community-based organizations (CBOs) that may choose to implement them in their jurisdictions.

We view our mixed efficacy outcomes from several perspectives. The fact that exposure to the app was associated with a doubling of HIV-testing behaviors and of prevalence of PrEP use among high-risk HIV-negative GBMSM suggests that the messages and app services could be a scalable, effective resource for GBMSM who engage in anal intercourse that is not protected by condoms or PrEP. Large national surveys of GBMSM suggest that 47%-53% of HIV-negative GBMSM would fall into this category [36,37]; considering the number of high-risk HIV-negative GBMSM in the United States [38], the potential user base for these services is about 1,800,000-2,000,000 US GBMSM.

The effect sizes for uptake of HIV screening and PrEP use were both about 2. However, the baseline levels of HIV screening and PrEP use were quite different. For high-risk GBMSM, the CDC recommends HIV testing at least annually, and in the intervention arm, nearly two-thirds reported HIV testing in the immediate posttest assessment period (tested in the prior 3 months). However, despite the same relative increase in PrEP use, only 1 in 6 (17%) high-risk GBMSM was taking PrEP at the 3-month postintervention time point. This suggests that additional and more intensive intervention services will likely be required for high-risk PrEP-eligible men who do not initiate PrEP within 6 months of starting the use of the Mobile Messaging for Men (M-cubed) app.

We also recognize that increases in PrEP usage and HIV testing associated with allocation to the app were short lived. The timing of the effects makes sense: the support for HIV testing provided during the period of app use included ordering at-home kits and navigating to testing locations; the significant increase in testing was observed during the period when men had access to the app, and about 1 in 7 high-risk men who tested for HIV used a mailed-out HIV self-test kit provided through the study. For PrEP, after the decision to start PrEP, there is a long process of making an appointment, undergoing lab tests, filling a prescription, and starting the medication; this outcome was higher in the 3 months after the period of app use. In practice, exposure to the app would not be limited to 3 months, and it is a separate question as to whether changes in behavior would be sustained over time with ongoing exposure to the app. Such a question could be answered in an implementation study.

Our approach was developed with an eye toward being able to reach large numbers of GBMSM with a basic package of prevention services. Given that there was efficacy for some outcomes, there are important questions about how the intervention might be made available to GBMSM who would be likely to benefit from it [39]. Most HIV prevention services are currently provided and funded through CBOs and health departments. Although some CBOs and health departments might have the technical capacity to support app-based prevention services, this capacity is likely quite limited. A currently underway randomized study of an efficacious eHealth intervention (Keep It Up [40]) will examine the outcomes of centralized versus decentralized implementation approaches, and the results of the study may inform how the M-cubed intervention could be most effectively implemented [41].

Our study was subject to several limitations. Although we used recruitment quotas to guide recruitment and achieved a balanced sample with respect to the 3 HIV status/risk groups, race/ethnicity, and recruitment venues, our sample was skewed to men with higher education. These selection biases threaten the external generalizability of the study; it is possible that the intervention might have had salutary effects for low-risk HIV-negative GBMSM or LWH GBMSM with lower education. However, although not powered for secondary comparisons, our post hoc stratified analyses did not suggest differential effects by educational attainment. Second, PrEP use in our sample was high at baseline (about 29% among all HIV-negative GBMSM), which is unique in large samples of GBMSM to date. For example, current PrEP use was reported by less than 20% of respondents to the American Men’s Internet Survey (a large, national sample of US GBMSM) in 2017 [10] and by 25% of GBMSM who participated in 23 cities in the 2017 National HIV Behavioral Surveillance study [37]. Over half of GBMSM in our study who were not on PrEP reported HIV testing in the 3 months before enrollment; this is similar to the proportion who tested for HIV in the 12 months before interviews in the 2016 AMIS survey [11]. Third, outcomes were self-reported and so might be subject to misclassification because of social desirability bias [42]. Fourth, our sample was restricted to GBMSM recruited online; this might introduce selection bias, in that men recruited through physical venues, such as sex venues, might have higher or lower levels of risk. Lastly, our sample was limited to GBMSM in 3 urban areas of the United States who reported sex with a man in the past year; patterns of risk behaviors and use of prevention services are known to be different in other urban areas or among rural GBMSM [11,21]. The level of ART use in our LWH MSM was high, offering limited opportunity for improvement. Future studies of the app could condition enrollment on the need for support for ART initiation or maintenance for LWH MSM.

The next steps for the study include secondary analysis of extensive app usage data or paradata [43] in terms of understanding the ratings of each message and the time spent using various optional aspects of the app descriptively [44]. We will also conduct further assessments of the potential mediation of outcome effects by app usage to help direct future prevention research and program implementation of mobile apps. Based on our data, the app offered benefits to GBMSM who reported recent anal sex not protected by condoms or PrEP but not to HIV-negative men who did not report sex not protected by condoms of PrEP or LWH men. Therefore, we will consider how the app can be updated and implemented [39,45] for GBMSM engaging in sex not protected by condoms or PrEP and for LWH MSM. Because we found that the app is a successful way to increase the intention to test for STIs and to distribute at-home STI self-test kits, the kits were often not returned and additional research is needed to understand the barriers to returning STI specimen collection kits. If this barrier can be overcome, the M-cubed app might also effectively support increased STI testing for some groups of GBMSM.

We sought to produce and evaluate a mobile app with prevention messaging and services to support comprehensive HIV prevention and care with low levels of prevention staff interaction for all GBMSM, regardless of HIV status or current risk behaviors. Such a serostatus-neutral approach, if implemented successfully, would offer advantages in terms of cost efficiency and minimization of segmentation of GBMSM by serostatus. Future work with the M-cubed app should focus on implementing the app for GBMSM who are likely to benefit from it based on their current behaviors and evaluating methods of implementation of the app through public or publicly supported prevention providers [39].