As anxiety disorders (ADs) increase worldwide [1-5], app-based interventions offer evidence-based, high-fidelity treatment options [6] with incredible potential and comparable effects with traditional treatment models in efficacy trials [7]. Several recent meta-analyses that assessed app-based interventions for anxiety have indicated a growing evidence base [7-9]. A recent 2021 Nature review (k=22) [10] found significant but small effect sizes on anxiety symptomatology (Hedges g=0.2888; P<.001). These interventions can fill an important treatment gap, mitigate the growing burden on providers [11,12], and reduce barriers to care (eg, time, cost, and stigma) [13]. However, app-based interventions for AD often have low levels of user engagement, which reduces their efficacy and inhibits broader health care implementation [14]. To capitalize on the benefits of these interventions, engagement needs to be effectively understood and analyzed [10]. Specifically, research identifying the optimal levels of engagement with app-based interventions and the effects of various app features on anxiety-related outcomes is critical to realizing their full potential as treatments for AD [15-17].

To date, research on engagement remains limited and is often not assessed, as app-based interventions are still largely analyzed using conventional methods (eg, intent to treat) [18]. This method evaluates the effect of assignment to treatment (ie, the effect of being randomized to a group) instead of the direct effect associated with the app features that were actually used during the intervention [18]. Analyzing which features the user engages with is critical to designing personalized, effective interventions [14] and allows researchers to understand which in-app features drive changes in health outcomes [14,19]. Understanding engagement is also important for app development; specifically, it contributes fundamental information toward identifying the optimal intervention dose [17], streamlining inefficiencies [20], and tailoring apps to specific clinical populations [21]. Recognizing the importance of information on engagement, research is increasingly prioritizing the investigation of the relationship between engagement and improvements in mental health outcomes [10,16,22,23]. Encouragingly, in one of the first meta-analyses examining the effect of engagement on health outcomes, Gan et al [24] (k=25) found significant moderate improvements in postintervention symptomatology for participants categorized as having higher engagement than participants with lower levels of engagement (Hedges g=0.40; SE 0.16; 95% CI 0.097-0.705; P=.01). Although only 5 studies were designed to target AD, the effects indicated a significant positive association between engagement and anxiety (r=0.33; 95% CI 0.24-0.41; P<.001) [24]. This work is promising; however, the field remains preliminary and requires additional research to enhance the growing understanding of the associations between engagement with app-based interventions and anxiety [25-27].

Inconsistency in the definition and measurement of engagement is an ongoing challenge associated with analyzing the construct within this promising preliminary research [24]. Engagement can be broadly conceptualized as “(1) the extent (eg, frequency, duration) of usage; and (2) a subjective experience characterized by affect, attention, and interest” [28], as it relates to subsequent changes in the targeted health behavior (eg, anxiety and depression) [29,30]. However, different industries (eg, psychology and marketing) have historically focused on specific parts of this definition (eg, in-app use vs levels of engagement in the targeted health behavior), often failing to fully capture an understanding of the complex relationships that make up engagement [29,30]. To improve our understanding of app-based interventions and health outcomes, it is critical to understand the relationship between what is used during the intervention and subsequent changes in health outcomes [31]. One model proposed by Cole-Lewis et al [31] addresses this relationship and posits that engagement is multifaceted, encompassing multiple definitions from various disciplines. The model links both system-level and behavior-level engagement [31]. They defined engagement as a multidimensional construct encompassing a user’s interactions with app features that influence specific behavioral determinants, resulting in increased engagement in the targeted health behavior [31]. The model posits that the use of in-app features (ie, microlevel engagement) is directly associated with changes in the desired health outcome (ie, macrolevel engagement). Understanding both microlevel and macrolevel measurements is necessary to evaluate the effects of app-based interventions on improvements in mental health symptomatology [31].

Applying this model of engagement, we conducted a secondary analysis using multivariate regression models to examine the associations between in-app engagement (ie, microengagement) and mental health outcomes (ie, macroengagement) for participants with generalized AD (GAD) using a targeted mobile app called Unwinding Anxiety (UA) [32]. Data were collected from the intervention group in a recently published randomized controlled trial (RCT) that tested UA versus treatment as usual [32]. The UA app is a theory-driven, multifaceted app that comprises both guided (ie, educational modules) and unguided features (eg, meditations and ecological check-ins) targeting novel reinforcement learning constructs [32,33]. The results from the primary RCT (N=65) were promising, with participants in the UA group reporting a median reduction in anxiety scores of 8.5 (IQR 6.5; P<.001) and the treatment as usual group reporting a median reduction of 1 (IQR 5; P=.01), representing a 67% versus 14% reduction at the 2-month follow-up [32].

The aims of this study were 2-fold: (1) to investigate whether higher microengagement with the primary intervention feature (ie, educational modules) is associated with positive changes in mental health outcomes at 2 months (ie, anxiety, worry, interoceptive awareness, and emotional reactivity) and (2) to investigate whether the use of adjunctive app features is also associated with changes in mental health outcomes. We hypothesized that higher levels of microengagement with educational modules would be associated with significant changes in outcomes, which is consistent with improved mental health.

This study is a secondary data analysis of a previously described stage 1 parallel-group RCT [32]. As this secondary analysis is interested in system use data and their association with study outcomes, data were taken from the intervention arm only, and study procedures relevant to analyzing the intervention group are summarized in the following sections [32]. Roy et al [32] provide detailed information on the study design, procedures, and the results of the randomized trial.

The primary trial was registered at Clinicaltrials.gov (NCT0368472), and the Brown University Institutional Review Board approved the study procedures (reference number PV4802) [32].

The data used for these analyses were from the baseline and 2-month time points for the intervention group only [32]. The inclusion criteria were as follows: (1) score ≥10 on the Generalized Anxiety Disorder 7-item scale (GAD-7), (2) owning a smartphone, (3) willingness to receive check-in calls, and (4) aged ≥18 years [32]. Participants were excluded from the study if they reported (1) dose changes of any psychoactive medication in the previous 2 months; (2) needed use of benzodiazepines and hypnotic sleep aids; (3) a history or current diagnosis of bipolar, schizophrenia or schizoaffective, or another psychotic disorder; (4) a significant medical condition that would affect the ability to complete study tasks; (5) cohabitation with someone already enrolled in the study; and (6) having a previous history of using other related apps, specifically Eat Right Now or Craving to Quit, which use similar reinforcement processes to UA [33-35]. Participants were recruited using social media largely through Facebook advertisements.

Eligible participants underwent informed consent procedures before enrolling in the study [32]. After enrollment, participants completed an in-person interview using the Mini International Neuropsychiatric Interview (MINI) International Neuropsychiatric Interview to confirm a diagnosis of GAD along with the assessment of other potential comorbid disorders (eg, depression, obsessive-compulsive disorder, and posttraumatic stress disorder) [32]. The participants were then asked to complete a web-based questionnaire using Qualtrics. Follow-up questionnaires were administered 2 months from treatment initiation using personalized email links specific to each participant’s unique identification number [32].

UA is an app-based intervention comprising educational modules that are considered the primary intervention features, consistent with recommendations from recent meta-analyses. The modules comprised instructional psychoeducational videos (5-15 minutes per day) teaching reinforcement learning concepts (Table 1). Modules are locked until the previous module is completed; however, participants can return to any of the already completed modules for review.

In addition, the app offers unguided adjunctive features divided into 2 categories: ecological features designed to synergize with skills learned in the modules and meditation practices. The ecological features included physiological check-ins and 2 types of stress evaluation: a meter that evaluates the strength and reason for stress or anxiety and a stress test that assists in familiarizing participants with practicing curiosity regarding stress or anxiety using interoceptive awareness skills learned in the program. Detailed descriptions of these features can be found in Table 1, a visual depiction of the main dashboard is shown in Figure 1 and an example of an adjunctive ecological feature is provided in Figure 2. The example is the psychological check-in feature, in which a participant is asked first to identify their current emotional state from an initial list and then rate their anxiety level at the moment, and it ends by offering a recommendation for a short practice to return to present moment awareness depicted from left to right.

For the adjunctive meditation features, participants had access to a series of 3 practices that they were encouraged but not mandated to use: Resting in Awareness, a body scan practice, and the Loving Kindness practice (Table 1) [36]. Each meditation had the option to choose from 4 lengths of time (7-30 minutes). The intent of offering varying lengths of time is that participants initially engage in shorter practices and progress to longer periods of sustained meditation. A dashboard with meditation features is shown in Figure 3. An example of meditation (ie, Loving Kindness) is depicted in Figure 4.

This was a secondary analysis of a previously conducted RCT. As such, the analysis was restricted to 32 participants who were randomized to the intervention group. A complete case analysis was conducted in this study. Of the 32 participants in the original intervention sample, 2 (6%) participants dropped out of the study before follow-up, and 2 (6%) participants did not complete the follow-up assessments, resulting in a final sample of 28 (88%) participants for this secondary analysis representing 88% of the original sample. For the primary RCT, a sample size of 52 was determined to have 80% power (1-sided Cronbach α=.05) to detect a statistically significant between-group difference in the primary outcome of anxiety (Cohen d=0.7). The final sample size was increased to 65 participants to account for 25% attrition.

Descriptive statistics, including means (SDs) for continuous variables and frequencies and percentages for categorical variables, were calculated for baseline variables, including demographics, anxiety, emotional regulation, and worry. Use data included ecological assessments (ie, check-ins, stress tests, and stress meters), mindfulness practices with the length of time completed (ie, Loving Kindness, Resting in Awareness, and body scan), and educational modules completed. Correlation analysis was used to identify potential confounders. Specifically, variables (eg, age, education, and income) were assessed using Spearman correlations, and those that met an a priori threshold of P≤.20 were included in subsequent models.

Microengagement was reported as the total number of times each app feature was accessed (ie, the total amount of use per item). Spearman rank correlation matrices were run to identify potential predictors to be included in the univariate linear regression models; predictors with a significance of P≤.20 were included [49]. Skewed engagement data were log transformed, except for the number of educational modules completed. Instead, the educational modules were dichotomized based on the completion of ≥75% (23/32) of the modules.

A series of multivariate linear regression models were constructed to investigate the dose-response relationships regarding the effect of use variables on psychosocial outcome measures. First, to determine predictors for inclusion in multivariate models, we ran a series of univariate linear models controlling for baseline scores to determine the impact of use variables on anxiety (ie, GAD-7), worry (ie, Penn State Worry Questionnaire), emotional regulation (ie, FFMQ), and interoceptive awareness (ie, Multidimensional Assessment of Interoceptive Awareness) at 2 months after treatment initiation. Predictors that were associated with outcomes at a modest P≤.20 level and met the assumptions testing criteria were then included in the subsequent multivariate linear regression models.

Final multivariate models investigated total meditation use, total ecological assessment use, individual meditation practices, and individual ecological assessments as predictors of the 4 outcomes (ie, anxiety, worry, interceptive awareness, and emotional regulation). All models controlled for baseline scores, as well as age and education level, as both have been shown to affect mobile app engagement in health behavior change in previous research [16]. Biological sex was not included as a covariate in the models to preserve parsimony because of a lack of correlation with outcomes (ie, P≥.20) in exploratory analyses and as only 7% (2/27) of the total sample identified as male versus 93% (25/27) as female. Analyses were run in R (version 4.0.3), and an a priori α level of .05 was set for the analyses. All models were evaluated to meet relevant model assumption criteria (eg, homoscedasticity and normality of residuals), and only models meeting the assumption criteria were reported. Summaries include R² statistics (measure of effect size), β coefficients, 2-tailed t test values, P values, and 95% CIs for all the included models.

The participant demographics are presented in Table 2. The sample was primarily female (25/27, 93%), and the average age was 42.9 (SD 15.6) years. Most participants reported completing some level of college education and self-reported as employed: 67% (18/27) reported having attained a bachelor’s degree or higher, 59% (16/27) reported having full-time employment, and 74% (20/27) reported an annual income of ≥US $40,000 per year.

We explored potential differences in demographic information across high versus low engagement with the main intervention feature—education modules. High versus low engagement was quantified as those who completed >75% of the education modules (≥23 modules; high completion) and those who completed <75% (<23 modules; low completion). Both the high and low engagement groups averaged consistent numbers of participants who reported psychiatric conditions (Table 2). With one exception, the high engagement group had significantly more past depressive episodes (P=.02).

The baseline anxiety, emotional regulation, worry, and interoceptive awareness scores are reported in Table 3. Overall, the sample had average anxiety scores of 13.0 (SD 4.9) and high average worry scores of 65.5 (SD 7.1). The average interoceptive awareness score was 78.7 (SD 22.4), and participants had an average low emotional regulation score of 15.2 (SD 4.2). No significant differences were observed between the high and low engagement groups in the primary outcomes at baseline.

Participants completed an average of 20.2 (SD 11.4) modules of the primary intervention feature for the total sample. Of these, 52% (14/27) were considered “high completers” (ie, ≥23 modules). Use data of the adjunctive features over the 2-month intervention period are presented separately for high versus low engagement in Table 4. For the ecological features, participants’ total engagement averaged 38.1 (SD 52.6) uses. Regarding specific ecological features, the average check-in use accounted for most of the use at 31.1 (SD 45), whereas stress tests and stress meters averaged <5 uses for the 2-month period. When assessing adjunctive ecological feature engagement across high versus low primary intervention engagement, the high engagement group used adjunctive ecological tools more across all types of features than the low engagement group (Table 4).

Participants averaged 7.2 (SD 7.2) mindfulness practice uses over the 2-month period. The range of uses varied for meditations, although not as broadly as ecological tool use, with some individuals never using the meditations and other participants using the meditations more frequently, with median meditation use of 7 (IQR 8). In addition, when assessing meditation use across intervention engagement categories (ie, high vs low), those who completed a high level of the primary intervention also used meditation practices more on average (Table 4). Specifically, those who had a higher engagement with the primary intervention also used significantly more meditations overall, as well as the Loving Kindness meditation and Resting in Awareness meditation (Table 4).

Summary statistics of meditation practice use for the full sample, and for high versus low engagement, are presented in Table 3. The completion of cumulative mindfulness practices across the entire sample averaged 7.2 (SD 7.2) over the 2-month period. The range of uses varied for meditations, although not as broadly as the ecological tool uses, with some individuals never using the meditations and other participants using the meditations more frequently, with median meditation use of 7 (IQR 8). In reviewing differences in average meditation use between high and low engagement, the group with high engagement also used meditation practices more on average, and significant differences were observed for cumulative meditations used, Loving Kindness, and Resting in Awareness but not for the body scan meditations (Table 4).

Comprehensive app-based interventions offer promising, high-fidelity treatments for AD, which are critical for addressing the growing treatment needs of the health care system [7,9,10,50]. To fully realize their potential, research will need to maximize engagement through a comprehensive understanding of how microengagement with in-app features affects macrolevel engagement in the target health outcome [31]. Our analyses sought to build on existing evidence by using multivariate linear regression models to examine the associations between improvements in mental health outcomes and the use of specific features within the UA mobile app. The results indicated that microengagement with app features was associated with significant changes in macroengagement in health outcomes (ie, interoceptive awareness, anxiety, emotional regulation, and worry), consistent with the engagement model proposed by Cole-Lewis et al [31]. Consistent with our hypothesis, engagement with the primary intervention feature (ie, completing at least 75% of the modules) was associated with an average increase of 22 points in interoceptive awareness scores and an 11-point decrease in clinical measures of worry at the 2-month follow-up. Associations were also found between the adjunctive app features (ie, ecological features and meditations) and improvements in health outcomes. Total meditation use was associated with a highly significant 0.62-point average reduction in anxiety (GAD-7), and the use of the Stress Meter, a tool for recognizing and investigating stressful situations in the moment, was associated with an average increase of 0.5 points in emotional regulation.

Moving forward, this research has significant implications for testing, developing, and customizing app-based interventions that target AD. First, this study offers important evidence that engagement with in-app features is a critical mechanism for increasing treatment benefits. Specifically, our findings showed a strong association between module completion and mental health outcomes, offering important evidence within the limited field of work on engagement with app-based interventions for AD. Previous research has largely focused on module completion as the primary engagement metric, finding small but significant associations with improvements in mental health outcomes for all digital interventions (eg, internet based and app based) [51-55]. However, these studies were mostly internet-based interventions, with few apps clinically tested for the effects of engagement on AD-related outcomes [23,24].

In addition, we found that intervention effects were driven by more than module completion, with findings indicating that adjunctive features (eg, ecological check-ins) were associated with important changes in health outcomes. The adjunctive or supportive components of complex app-based interventions have rarely been investigated. Instead, to date, most research has favored the analysis of completed modules or using a total frequency metric for all features used [24]. Focusing singularly on these adjunctive components offers 2 important insights: a clear indication of how those additional features are associated with changes in outcomes and information on potentially unnecessary or ineffective tools, which may needlessly complicate the app. Future research building on this study, focusing on understanding how specific supportive features function mechanistically to affect clinical outcomes, will be important as this information allows developers to streamline digital interventions [31]. By prioritizing streamlining app-based interventions, highlighting the effective features and removing ineffective features we can increase intervention efficiency as well as the likelihood of initiating and sustaining health behavior changes [22,56].

Although our analyses offer important novel insights into engagement and health outcomes, to fully capitalize on the potential of app-based interventions for sustained behavior change engagement, research needs to move beyond considering engagement as an aggregated total. Summed user engagement metrics, such as those used in these analyses, offer important insights but provide a limited view when considering the practical reality of engagement as a dynamic interaction between the participant and the app over time [22,23,47,57]. In the case of UA specifically, the app is designed to develop knowledge (eg, retraining reinforcement learning, education on understanding anxiety, and goal-directed behavior training) and promote skills maintenance (eg, meditation and emotional regulation), which can be applied in everyday life. The eventual goal of the app is actually “off ramping” participants or effectively reducing microengagement with the app while sustaining the associated health outcome improvements (eg, reductions in anxiety and increased emotional regulation). In this case, the pattern of engagement may change; for example, a participant may start at a higher level of engagement but begin to titrate how often they use the app over time. As such, the goal for UA is that health behavior change would be sustained despite lower engagement over time, which cross-sectional or sum engagement metrics cannot reflect. Thus, future analyses will need to move beyond aggregated engagement and analyze individual time series data to capture the dynamic, longitudinal relationship between in-app engagement and sustained health behavior changes.

Although the analysis presented is the first to investigate associations between the use of UA tools and mental health outcomes, the underlying reinforcement learning theory and curriculum for the primary intervention feature have been associated with behavioral reductions in our previous work on smoking cessation [58,59]. Specifically, our findings regarding module engagement build on results from an RCT (n=225) of the Craving to Quit program, a mobile platform designed to help people quit smoking, which uses a similar suite of educational modules grounded in the reinforcement learning theory and application of meditation practices on which the UA program is founded. These previous results indicated that educational module engagement, categorized as low (0-14 modules), medium (15-41 modules), and high (≥42 modules) in linear mixed models, was associated with significant reductions in the relationship between craving and cigarette smoking at 6 months (F_1,104=4.44; P=.04) in the intervention group (n=182) [58].

In addition, in a previous trial of an app-based intervention targeting eating behaviors and craving-related eating—Eat Right Now—the results indicated significant associations between ecological feature engagement and craving-related eating [60]. Eat Right Now also comprises ecological features, which guide participants by investigating cravings, in addition to mindful eating exercises, to reduce the reward value of overeating [60,61]. The results showed significant effects of ecological feature use on reductions in craving-related eating, anticipated reward value of craving-related eating, and the likelihood of engaging in craving-related eating within an initial sample of 46 participants, which was then replicated among a larger sample (n=1119) using Rescorla-Wagner computational modeling [61].

This study, coupled with previous work, demonstrates the important role of multiple engagement metrics associated with positive changes in health outcomes among a range of clinical populations.

This study has some limitations. First, engagement data were calculated as the total number of uses. It is possible that a feature was opened, closed, or stopped midway. It is also possible that the participant was not actively engaged with the feature (eg, distracted or the participant was not the one using the app). In future studies, the length of use time should be considered for analysis to mitigate these limitations. In addition, this analysis considered specific features that operate singularly on health outcomes. However, this is unlikely to reflect the engagement patterns of participants who use a combination of features in pursuit of their health goals. Aggregation of microengagement patterns with the inclusion of the total time of engagement may offer windows into specific “usage profiles,” which could more effectively reflect the interdependent nature of many of these features. Another important limitation was the small sample size available for analysis. In addition, the sample was overwhelmingly female, White, and highly educated, making the generalizability of these findings to broader populations questionable. The length of follow-up may also have been a limiting factor for the analysis as there was variability regarding module completion, which may not have been aligned with the 2-month assessment window. Future analyses should use larger, more diverse samples to test the impact of microengagement patterns in the broader population.

This secondary analysis offers evidence of associations linking the use of in-app features in UA to improvements in mental health outcomes; however, these results are preliminary and exploratory. The work presented offers a clearer understanding of the impact of how microengagement with the app features affects macroengagement with health outcomes of interest, consistent with the model proposed by Cole-Lewis et al [31]. This study highlights the importance of comprehensive investigations of engagement during the development of evidence-based app-based interventions. Future research would benefit from comprehensive, longitudinal analyses, as well as primary studies that specifically assess the effects of varying levels of engagement trials on health outcomes (eg, microrandomized trials, rapid optimization methods, and multiphase optimization strategies) [18,62-64]. As research continues, understanding the effects of engagement with app-based intervention features on clinical outcomes will be critical to designing targeted interventions needed to increase patient health and support accessible, comprehensive care.