Young sexual and gender minority men who have sex with men (YSGMMSM) are burdened with a disproportionate and growing vulnerability to HIV in the United States [1-8]. Of the 30,692 incident HIV cases in 2020, 71% were among men who have sex with men (MSM) and 24% were among MSM aged 13 to 24 years [8]. In addition, a meta-analysis of transgender women aged ≥15 years found that they were 48 times more likely to have HIV infection compared to other adults of reproductive age [1]. While pre-exposure prophylaxis (PrEP) has demonstrated efficacy in reducing the incidence of HIV infections in YSGMMSM in the US in randomized controlled trials (RCTs) [9,10], adherence to these medications outside of clinical trial settings has been suboptimal for reducing transmission [9,11]. Furthermore, findings from population and agent-based simulation studies demonstrate that PrEP uptake and adherence are associated with a decrease in incident HIV infections, lowering incidence by as much as 25% [12,13]. Altogether, this suggests that improving PrEP adherence is a fruitful pathway to reducing HIV incidence in YSGMMSM [9,11-13].

Digital health interventions (DHIs) are potentially powerful and increasingly prevalent mechanisms for delivering PrEP interventions to YSGMMSM [14-20]. Due to the pervasiveness of digital communication and entertainment among youth, including YSGMMSM, DHIs are suitable delivery mechanisms for HIV prevention interventions due to their potential to effectively engage participants [21-25]. However, to our knowledge, there is a paucity of clinical trials directly testing the efficacy of digital PrEP adherence interventions. Adjacent digital PrEP adherence interventions, such as automated SMS text-messaging services or digital pill systems, have had mixed results in young populations. For example, digital pill systems have documented barriers to engagement and have not shown that they independently improve PrEP adherence compared to a standard of care [26-29]. Furthermore, DHIs implemented to address other health problems in target youth populations have encountered problems in effectively engaging the participants. For example, 1 study that developed a mobile app for self-management of type 1 diabetes among adolescents found no association between intervention conditions and primary or secondary clinical outcomes. Furthermore, the study found that only 9% of the participants met the criteria for high engagement levels (measured as an individual uploading blood glucose readings for ≥3 d/wk) [30]. For these reasons, effectively engaging YSGMMSM in PrEP adherence DHIs is likely an essential element to achieving protective levels of PrEP adherence [25,31,32].

The effective engagement framework by Yardley et al [25] defines effective engagement as “sufficient engagement with the intervention to achieve intended outcomes.” They describe four phases of engagement as follows: (1) initial engagement with the DHI in preparation for behavior change; (2) engagement with the behavior change, mediated by the DHI; (3) DHI use may no longer be required to sustain behavior change; and (4) reengagement with DHI as needed. Furthermore, this framework describes how, in phase 1, effective engagement is largely characterized by a micro form of engagement (defined by moment-to-moment interactions with the DHI). As individuals progress through phases 2, 3, and 4, they are increasingly engaged with a macro form of engagement (defined as engagement that relates to the overarching goal of the intervention).

The effective engagement framework by Yardley et al [25] also highlights how intrapersonal, social, and environmental characteristics can influence effective engagement and how tailoring DHIs based on those characteristics may improve effective engagement. Tailoring, defined as the use of individuals’ data to customize intervention content based on their psychological, socioecological, and behavioral profile, is a property of DHIs that has demonstrated efficacy in promoting effective engagement with DHIs [25,33-37]. Several barriers and facilitators to engagement have been identified in qualitative research [38]. For example, baseline motivation to change [25,39] or baseline comfort with the intervention modality (eg, phone or app-based intervention) may influence engagement and subsequent behavior change [38-49]. Furthermore, aspects beyond the individual’s control, such as internet access, may influence engagement and therefore intervention efficacy [38,50-52]. However, there is a lack of quantitative research that aims to isolate and quantify the impact that intrapersonal psychological, sociodemographic, and behavioral measures have on engagement and subsequent behavioral outcomes of interest.

Yardley et al [25] and other review articles using their effective engagement framework describe a plethora of issues with quantitively operationalizing and measuring effective engagement in previous research. Most notably, engagement research to date is largely correlational, relies on the assumption that engagement is intrinsically a precursor to the intended outcome, and does not account for intrapersonal measures, such as motivation to use the intervention or digital literacy [25,31,32,39,53-55]. This highlights the need to empirically test and corroborate the models of engagement, by modeling how intrapersonal measures influence engagement, which subsequently mediates behavioral outcomes of interest (eg, PrEP adherence) [25,53,56]. Causal mediation analysis is a fitting statistical framework for investigating the relationship between intrapersonal measures and effective engagement which addresses the aforementioned methodological issues as well. Causal mediation has been extensively explicated [57-62], applied in previous research in other domains [63], and addresses the aforementioned issues by Yardley et al [25] by clarifying assumptions needed for causal inference in mediation models, explicitly modeling the relationship between engagement and outcome (as opposed to relying on the assumption that engagement and the outcome are related), and incorporates intrapersonal measures (as exposures or controls) [57,58,60]. This approach is derived from the counterfactual causal inference framework [64,65] and allows for the total effect (eg, the effect of baseline digital literacy on PrEP adherence) to be decomposed into a direct and indirect effect. The direct effect models the effect of a given intrapersonal measure on PrEP adherence controlling for engagement with the intervention. Conversely, the indirect effect models how a given intrapersonal measure is related to PrEP adherence operating through engagement, and thus, is an excellent measure of effective engagement. The integration of the effective engagement framework with the casual mediation approach provides a combined theoretical and analytical approach for evaluating effective engagement in DHIs.

This study collates clinical survey data, biological PrEP adherence measures, and engagement measures from the Prepared, Protected, Empowered (P3) intervention efficacy RCT [19,66] in a secondary data analysis using the causal mediation framework [57-62] to quantify whether and to what degree intrapersonal behavioral, mental health, and sociodemographic measures impact effective engagement with respect to PrEP adherence in YSGMMSM.

This study is a secondary analysis, which combined clinical survey data, biological PrEP adherence measures, and engagement measures collected from a 3-arm RCT testing the efficacy of P3, a PrEP adherence DHI [19,66]. This secondary analysis used the causal mediation framework and statistical analysis procedures to characterize whether engagement with the P3 intervention mediated the relationship between baseline intrapersonal measures and PrEP adherence at 3 months.

The parent study was reviewed and approved by the institutional review board of the University of North Carolina at Chapel Hill (17-9551). A certificate of confidentiality was obtained from the Eunice Kennedy Shriver National Institute of Child Health and Human Development. For participants aged between 15 and 17 years, a waiver of parental consent was obtained. This trial is registered at ClinicalTrials.gov (NCT03320512). This secondary analysis was reviewed by the Northeastern University institutional review board and determined to be exempt under category 4 (secondary research for which consent is not required). This secondary analysis used a deidentified analytic dataset curated by the parent study’s staff. The principal investigator had no contact with participants and made no attempts to reidentify participants post hoc.

Participants from the primary study who received the P3 or P3+ intervention were eligible for inclusion in this secondary data analysis (n=163). Participants in the P3 and P3+ conditions who were lost to follow-up (LTFU; defined as participants who did not begin the month 3 survey) were excluded (22/163, 13.5%). Moreover, 1 (0.7%) of the remaining 141 participants with incomplete survey information pertaining to study-relevant exposure measures was also removed. This resulted in a dataset of 140 participants (Figure 1).

PrEP nonadherence at 3 months (binary) was the primary outcome measure used in this analysis. If serum levels of tenofovir-diphosphate and emtricitabine-triphosphate were consistent with ≤4 doses per week, the participant was considered nonadherent. Due to study operation interruptions related to the COVID-19 pandemic, 28.6% (40/140) of eligible participants were unable to provide biological specimens. In cases where tenofovir-diphosphate and emtricitabine-triphosphate values were missing, self-reported doses of PrEP in the last 7 days before the 3-month clinical survey were used. Participants who reported ≤4 self-reported doses of PrEP in the last 7 days were considered nonadherent. There is mixed evidence regarding the accuracy of self-reported measures of PrEP adherence. While 2 studies have found that self-report measures of PrEP adherence correlate with protective serum levels among adults [69,70], another study examining self-report PrEP adherence measure accuracy among 15 to 23-year-old young MSM found that self-reported measures overreported adherence compared to biological specimens, with the odds of overreporting decreasing by 24% (odds ratio [OR] 0.74, 95% CI 0.65-0.90) with each additional year of age [71]. We found that the area under the receiver operating characteristics curve between self-report measures and biological measures among participants with biological and self-report PrEP nonadherence measures was high (≥0.7). Further details are described in the Strengths and Limitations section.

Engagement was defined as dollars accrued by 3 months. This measure was mean centered (participant dollars accrued—mean dollars accrued) when used in modeling. As mentioned in the Intervention subsection, participants started with a baseline amount of US $90 in a bank and gained or lost money from this initial bank based on the number of days each participant logged in and completed one of the 3 aforementioned tasks (eg, completing a quest). This measure serves as a quality proxy for engagement because it correlates with the behavior pattern P3 aims to adjust (ie, daily use of P3 mimics the daily dosing pattern of PrEP) and relates to the putative behavior change mechanisms used in the P3 app (eg, completing a quest is related to gamification and posting on the social wall is related social engagement). Engagement measures derived from documenting users’ access, participation, and navigation through a DHI are suitable for constructing measures of engagement, as these measures provide an objective view into patterns of use with high ecological validity [25,31,32].

Consistent with Yardley et al [25], several intrapersonal measures capturing experiences with phone and phone apps, mental health, and sociodemographic characteristics were constructed from the baseline survey administered in the primary RCT based on prior research [25,38]. These measures, described subsequently, focused on patterns of phone and phone app use, measures of mental health, and sociodemographic information.

Descriptive statistics for participants who were eligible for secondary analysis were generated and are reported in the Results section. There is also a comparison of 140 eligible participants to the 22 participants who were LTFU. To quantify how engagement with P3 mediates the relationship between baseline intrapersonal measures and PrEP nonadherence at 3 months, the causal mediation framework was used. This analytical approach extends the counterfactual causal inference framework to mediation, has been extensively explicated, clarifies several confounding assumptions, accommodates exposure-mediator interaction [57-62,93], has been applied in previous research [63], and complements the effective engagement theoretical framework by explicitly decomposing the total effect of each intrapersonal measure on PrEP nonadherence into a direct and indirect effect (ie, the effect mediated by P3 engagement). Figure 2 depicts the integration of effective engagement and causal mediation through a theoretical causal diagram. In this study, effect decomposition was accomplished by first fitting a linear regression to assess the effect of each exposure (eg, anxious symptoms) on the mediator (mean-centered dollars accrued at 3 months), adjusting for confounding. Then, a logistic model was fit to examine the relationship between each exposure (eg, anxious symptoms) and PrEP nonadherence, adjusting for the same confounders, dollars accrued at 3 months (ie, the mediator), and the exposure-mediator interaction. Baseline measures, such as age, race, and intervention arm, were used to adjust for confounding. Several other critical measures were controlled through the primary study’s design via the eligibility criteria. Intention to initiate PrEP, PrEP access, sexual orientation, and English literacy were verified at enrollment. The total effects estimated from these 2 models with corresponding CIs and P values are reported in the Results section. The total effect for each intrapersonal measure on PrEP nonadherence at 3 months is presented as an OR (defined as the rate of nonadherence in the exposed divided by the rate of nonadherence in the unexposed). These 2 regression models are then used to derive direct and indirect effects. The direct effect represents the effect of a given intrapersonal measure on PrEP nonadherence independent of P3 engagement. This effect is estimated by comparing the estimated PrEP nonadherence in the exposed relative to the unexposed while setting P3 engagement to the level that would have naturally occurred in the absence of the exposure. The indirect effect represents the effect of a given intrapersonal measure on PrEP nonadherence operating through the mediator. This effect is estimated by comparing the outcome for the exposed for different contrasts of the mediator (eg, between levels of P3 engagement). Mediation results, including direct and indirect effects with corresponding CIs and percentage mediated (PM) are reported in the Results section. Mediation is assessed using a combination of total and indirect effect size, statistical significance, and PM. P values are reported for transparency, but CIs are used as the primary determinant of statistical significance for mediation analysis, determined by if the CI overlaps with the null value. For the first model (linear regression of dollars accrued), the null value is 0, and for the second model (logistic regression of binary PrEP adherence), the null value is 1. All analyses were carried out using R (GNU) and RStudio (Posit, PBC). Mediation models were constructed using the CMAverse R library using the regression-based approach and imputation as the estimation method [94].

The median age of participants was 22 years (IQR 20-23). Overall, 17.1% (24/140) were considered nonadherent to PrEP at 3 months (Table 1). The minimum amount the participants earned over the 90-day trial period was US $1.5 (representing 1 day logged in with a task completed). The maximum amount the participants earned over the 90-day trial period was US $135 (corresponding to logging in and completing a task every day for the 90-day trial period). Participants earned a mean of US $96.40 (SD US $35.1) over the 90-day trial period. Participants earned a median of US $112.50 (IQR US $73.50-US $ 123.50) which corresponds to 75 days logged in with a task completed (IQR 49-82) over the 90-day trial period. Participants were largely heterogeneous with respect to phone, technology, and internet use patterns. Notable exceptions to this pattern were as follows: 90.7% (127/140) of the participants used phone apps more than once per day and 90.7% (127/140) of the participants disclosed that they often use phone apps for chatting with friends. No significant differences in selected baseline intrapersonal characteristics were observed between eligible and LTFU participants (Table 1).

Table 2 presents multivariate regression results for the effect of intrapersonal measures on engagement (dollars accrued by 3 months). Frequent phone app users earned US $21.49 (95% CI 2.50-40.47) more than infrequent phone app users through the 3-month trial period. Participants who reported anxious symptoms in the past 2 weeks (GAD-7 score≥10) earned US $15.95 (95% CI –31.57 to –0.32) less throughout the 3-month trial period than those with mild or no anxious symptoms in the past 2 weeks. Non-Hispanic White individuals earned US $17.02 (95% CI 5.95-28.10) more on average by 3 months than participants belonging to other racial and ethnic groups. Participants who received the P3+ intervention (P3 with the addition of human adherence counselors accessible through the app) earned US $12.48 (95% CI 1.47-23.50) more on average than those who received the standard P3 app through the 3-month trial period. Finally, for each additional year of age, participants earned on average an additional US $2.55 (95% CI –0.19 to 5.29). The 95% CI for the relationship between age and engagement narrowly overlaps the null value of 0 with a corresponding P value of .07, meaning this relationship is technically statistically insignificant.

Multivariate models are adjusted for age, race, ethnicity, intervention arm, dollars accrued at 3 months, and dollars accrued at 3 months and the interaction between dollars accrued at 3 months and the focal intrapersonal measure.

Total effects for the relationship between baseline intrapersonal measures and PrEP nonadherence at 3 months are reported in Table 2 as ORs. For every dollar earned above the mean throughout the 3-month trial period, participants had 2% (OR 0.98, 95% CI 0.97-0.99) lower odds of PrEP nonadherence at 3 months. Participants who reported using phone apps ≥2 times per day had 34% lower odds of PrEP nonadherence at 3 months (OR 0.66, 95% CI 0.17-2.88). Participants who spent >7 hours on the internet beyond work or school had 2.31 (95% CI 0.80-6.46) times higher odds of PrEP nonadherence at 3 months compared to participants who reported <7 hours on the internet beyond work or school. Both of these measures that aimed to describe broad patterns of phone and internet use had a relatively large effect on the odds of PrEP nonadherence. However, neither was statistically significant as the 95% CI covers the null value of 1. Participants who reported at least 1 disconnect from their internet service or phone in the past year had 3.84 (95% CI 1.14-12.81) times higher odds of PrEP nonadherence at 3 months than participants who reported no disconnects. Participants who reported that they frequently used phone apps to track their personal health information had 71% (OR 0.29, 95% CI 0.06-0.96) lower odds of PrEP nonadherence at 3 months. Participants who reported depressive symptoms (PHQ-8 score ≥10) had 3.42 (95% CI 0.95-12) times higher odds of PrEP nonadherence at 3 months. Participants who reported anxious symptoms also had 3.51 (95% CI 1.06-11.55) times higher odds of PrEP nonadherence at 3 months. Participants who reported their race and ethnicity as non-Hispanic White had 83% (OR 0.17, 95% CI 0.05-0.48) lower odds of PrEP nonadherence at 3 months. For each additional year of age, participants had 18% (OR 0.82, 95% CI 0.65-1.02) less odds of PrEP nonadherence at 3 months. The 95% CI for the relationship between age and PrEP nonadherence narrowly overlaps the null value of 1 with a corresponding P value of .08, meaning this relationship is technically statistically insignificant.

Total effects were decomposed into direct and indirect effects of intrapersonal measures on effective engagement (Table 3). Experiencing disconnects in the past year was primarily directly related to higher odds of PrEP nonadherence at 3 months (PM=5%), with a direct effect of 3.28 (95% CI 0.91-11.42). Despite the statistically insignificant total effect, using phone apps ≥2 times per day was significantly indirectly associated with lower odds of PrEP nonadherence at 3 months (OR 0.76, 95% CI 0.49-1.00). Conversely, spending >7 hours on the internet beyond work or school resulted in primarily a direct relationship with lower odds of nonadherence (1.92, 95% CI=0.80-5.42; PM=18%). However, similar to the total effect on PrEP nonadherence, this relationship did not rise to the level of statistical significance for either the direct or indirect relationship. Frequently using phone apps to track health information was directly associated (0.3, 95% CI 0-0.92; PM=1%) with lower odds of PrEP nonadherence. Symptoms of depression had a statistically significant total effect on PrEP nonadherence and were partially mediated by engagement with P3 with a direct effect of 2.52 (95% CI 0.79-6.81), an indirect effect of 1.25 (95% CI 0.78-2.14), and a PM of 30%. However, neither the direct nor the indirect effect raised to the level of statistical significance on their own. Experiencing anxious symptoms was primarily indirectly related to higher odds of PrEP nonadherence through engagement with P3 (1.55, 95% CI 1-3.34; PM=51%). Being non-Hispanic White was directly related to lower odds of PrEP nonadherence (0.20, 95% CI 0.04-0.59; PM=4%). For each additional year of age, the odds of PrEP nonadherence were decreased, operating through both direct (0.95, 95% CI 0.68-1.02) and indirect (0.92, 95% CI 0.59-1.07; PM=60%) relationships.

This study has several methodological and analytical strengths. The research design of the primary RCT is complementary to the theoretical constructs of effective engagement and causal mediation analysis. RCTs provide clear temporality, which is simultaneously necessary to establish effective engagement (ie, engagement leading to a downstream health outcome) and causal mediation analysis [25,57-60]. This study demonstrates the clarity causal mediation analysis provides to engagement research by disentangling total effects into direct and indirect effects, allowing a causal and mechanistic characterization of how baseline intrapersonal measures relate to effective engagement. This approach helps to avoid mischaracterizations that can occur with traditional regression techniques, as these techniques often model engagement as the outcome and rely on the assumption that an increase in engagement would precipitate an increase in the behavior change of interest. For example, quantitative research into medication adherence disparities has found disparities exist among racial or ethnic groups after adjusting for other socioeconomic confounders [107], and qualitative research into engagement with DHIs has largely lacked a focus on race and ethnicity [38], which suggests that race and ethnicity might play a role in effective engagement for adherence DHIs. In this study, non-Hispanic White participants earned significantly more money than other trial participants (US $17) 3 months into the trial period. However, effect decomposition demonstrated that the effect of race and ethnicity on PrEP nonadherence was largely direct, suggesting that race and ethnicity is related to PrEP adherence irrespective of the observed increase in dollars accrued (ie, engagement) among non-Hispanic White participants. This example illustrates how traditional regression and causal mediation approaches would arrive at divergent conclusions based on the same data. Traditional approaches assume engagement leads to a behavior change, whereas causal mediation analysis estimates behavior change based on changes in engagement. In this example, given the results from the causal mediation analysis, it seems substantially more likely that reported non-Hispanic White race and ethnicity is a proxy for the systematic inequalities people of sexual, gender, racial, and ethnic minority face in the United States and that these inequalities propagate difficulties with PrEP adherence in ways P3 cannot or did not address.

One limitation of this study is the small and selective sample size. The small sample size limits the power to detect effects. This means that some of the null results in this study may in fact be more significant in a study with a larger sample. However, this also means that this study is likely a conservative measure of effective engagement. The significant results from this study which have relatively small effect sizes may in truth be much larger. Furthermore, 13.5% (22/163) of the eligible participants were LTFU, which is consistent with the primary P3 efficacy RCT, which observed 13% LTFU across all intervention conditions. Due to this, we believe that participants were not LTFU for reasons specific to study operations or intervention conditions. Similarly, the sample of individuals in the primary RCT is relatively homogenous (small age range, relatively high digital literacy, generally nonrural, and same sexual orientation). Therefore, the results of this study may or may not be generalizable to other populations. However, while this feature of the primary RCT limits generalizability, it also strengthens confounding assumptions necessary to carry out causal mediation analysis. By having a more homogenous sample, many of the measures that may have been conceived as confounders have been controlled for through the primary RCT’s study design, such as sexual orientation and English literacy. Furthermore, while several aspects of the population are homogenous, there is a high degree of geographic diversity as the primary RCT was carried out at 9 study sites. Another limitation is the degree of missingness in the biological measures of adherence. Before the COVID-19 pandemic, participants completed study activities in person at study sites, which included the collection of biological specimens by study site staff. Once COVID-19 restrictions were in place and study activities resumed, most activities were completed remotely to enable the continuation of data collection. Participants were asked to complete the at-home dried blood spot collection. The change from site-directed biological specimen collection to dried blood spot self-collection likely impacted the amount of missing biological specimen data. However, self-report measures are accurate for estimating protective serum levels of PrEP among adults [69,70] and the likelihood of incorrectly estimating protective serum levels of PrEP decreases significantly with age among adolescents and young adults [71]. Given that the average age of participants was 22 years in this study, combined with the relatively high area under the receiver operating characteristics curve (≥0.7) between self-report measures and biological measures among participants without missing biological measures, using self-report PrEP adherence measures where biological measures are missing seems sufficient. Finally, the inclusion of a validated digital literacy scale and social anxiety–specific scale (as opposed to only a scale of generalized anxiety disorder) would have been ideal to measure digital literacy and social anxiety, respectively.

This study used a causal mediation approach using secondary data from an RCT testing the efficacy of P3, a digital PrEP adherence intervention. This study combined data from the primary RCT, including biological PrEP adherence measures, with engagement data, to characterize how baseline intrapersonal measures relate to effective engagement in participants who received P3. Broadly, P3 engagement (dollars accrued) was strongly related to lower odds of PrEP nonadherence. Specifically, this study identified digital literacy as a potential engagement facilitator and measures of structural disparity (eg, disconnection from phone or internet in the past year) and mental health (eg, anxious symptoms) as engagement barriers. Study results suggest tailoring as a critical DHI mechanism to address barriers to engagement and emphasize engagement facilitators in indicated individuals. Furthermore, these findings highlight the suitability of causal mediation analysis for effective engagement research by delineating the total effect of each intrapersonal measure into direct and indirect effects (effective engagement). Future research into effective engagement would benefit from adopting a causal mediation approach. Furthermore, as hypotheses regarding exact mechanisms for fostering engagement arise, future research should measure engagement with measure-specific areas of the DHI as a mediator.