This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Meditation apps have surged in popularity in recent years, with an increasing number of individuals turning to these apps to cope with stress, including during the COVID-19 pandemic. Meditation apps are the most commonly used mental health apps for depression and anxiety. However, little is known about who is well suited to these apps.
This study aimed to develop and test a data-driven algorithm to predict which individuals are most likely to benefit from app-based meditation training.
Using randomized controlled trial data comparing a 4-week meditation app (Healthy Minds Program [HMP]) with an assessment-only control condition in school system employees (n=662), we developed an algorithm to predict who is most likely to benefit from HMP. Baseline clinical and demographic characteristics were submitted to a machine learning model to develop a “Personalized Advantage Index” (PAI) reflecting an individual’s expected reduction in distress (primary outcome) from HMP versus control.
A significant group × PAI interaction emerged (
Overall, the results revealed the potential of a data-driven algorithm to inform which individuals are most likely to benefit from a meditation app. Such an algorithm could be used to objectively communicate expected benefits to individuals, allowing them to make more informed decisions about whether a meditation app is appropriate for them.
ClinicalTrials.gov NCT04426318; https://clinicaltrials.gov/ct2/show/NCT04426318
Precision medicine, which involves the use of individual variability to guide prevention and treatment, has gained momentum in the health sciences over the past several years [
Psychiatry and clinical psychology have long hoped to better match patients with interventions. Numerous studies have examined patient-level factors (eg, demographic, clinical, and neurobiological characteristics) as predictors of treatment response [
Machine learning has emerged as a promising analytical approach well suited for handling and integrating large numbers of predictor variables, including correlated predictors, that may individually only modestly predict outcomes of interest but can collectively predict significant variance in patient outcomes [
In pursuit of precision mental health, researchers have leveraged machine learning approaches to optimize treatment recommendations [
Prior research using the PAI and related approaches [
A recent analysis of available mental health apps revealed that meditation and mindfulness training (along with journaling and mood tracking) are the most common features offered across apps [
This study involved a secondary analysis of a large-scale randomized controlled trial (RCT) comparing a meditation-based smartphone app, the Healthy Minds Program (HMP), with an assessment-only control condition [
Wisconsin school district employees were recruited via email and other electronic media between mid-June 2020 and late August 2020 (for a full description of study procedures, refer to the study by Hirshberg et al [
The trial was preregistered at ClinicalTrials.gov (NCT04426318) and through the Open Science Framework [
The HMP includes contemplative practices designed to build skills supportive of 4 pillars of well-being: awareness, connection, insight, and purpose [
The study procedures were approved by the University of Wisconsin—Madison Institutional Review Board (number 2020-0533).
The participants reported their age, gender identity, race and ethnicity, marital status, and income at baseline.
The prespecified primary outcome in the parent RCT was psychological distress, which was a composite of the computer-adaptive versions of the PROMIS Anxiety and PROMIS Depression measures [
Several additional self-report questionnaires assessed secondary outcomes and candidate mediators that were theoretically linked to the pillars of well-being trained within the HMP. The 15-item Perseverative Thinking Questionnaire (PTQ) [
Predictor variables included preintervention distress (composite measure), anxiety (PROMIS), depression (PROMIS), stress (Perceived Stress Scale), repetitive negative thinking (PTQ), the mindfulness facet of acting with awareness (Five Facet Mindfulness Questionnaire), loneliness (National Institutes of Health Toolbox Loneliness), defusion (Drexel Defusion Scale), presence (MLQ), search for meaning (MLQ), self-compassion (Self-Compassion Scale Short Form), well-being (5-item World Health Organization), age, gender, race, marital status, and income.
Missing data were imputed using a random forest–based imputation (MissForest package in R [
To predict outcomes, 2 prognostic models (using elastic net regularized regression [ENR]; glmnet package in R) were developed: one for participants who received HMP and one for those who received the assessment-only control condition. To minimize overfitting, which can occur with traditional
The steps mentioned earlier generated predicted HMP outcomes for HMP participants and predicted control condition outcomes for the control participants. To generate predicted outcomes for the counterfactual condition (ie, the treatment condition one did not receive), an ENR model was developed for one group (ie, full HMP or control sample) and used to predict outcomes for participants in the other group.
As a final product of the prediction models mentioned earlier, every participant had 2 predicted outcome scores: one for HMP and one for the control condition. Consistent with previous similar studies [
We compared the abovementioned multivariable machine learning (ENR) model with a simple linear regression with baseline repetitive negative thinking (PTQ) scores as the sole predictor (ie, repeating the above steps to generate a PAI score for every participant) implemented via 10-fold CV (repeated 100 times to generate stable estimates). Repetitive negative thinking was selected as a predictor in this comparison model based on prior research, indicating that it predicts response to mindfulness apps [
All analyses were conducted using R software (version 4.0.2) [
The majority (523/662, 79%) of the participants reported depression or anxiety symptoms at baseline that were above the clinical cutoff on the PROMIS Depression and PROMIS Anxiety measures (
The groups did not differ at baseline in terms of the demographic or clinical variables (
Descriptive statistics for Healthy Minds Program and assessment-only control at baseline.
Variable | Healthy Minds Program | Control | ||||||||
|
Value, N | Value, n (%) | Value, mean (SD) | Value, N | Value, n (%) | Value, mean (SD) |
|
|||
Age (years) | 344 | — | 42.47 (11.06) | 318 | — | 42.70 (10.23) | .78 | |||
Gender (female) | 344 | 299 (86.9) | — | 318 | 279 (87.7) | — | .75 | |||
Non-Hispanic White | 344 | 304 (88.4) | — | 318 | 268 (84.3) | — | .13 | |||
Married | 344 | 243 (70.6) | — | 318 | 216 (67.9) | — | .45 | |||
College education | 343 | 308 (89.8) | — | 316 | 281 (88.9) | — | .72 | |||
|
||||||||||
|
≤50,000 | 344 | 56 (16.3) | — | 318 | 55 (17.3) | — | .73 | ||
|
50,000-100,000 | 344 | 141 (41.0) | — | 318 | 129 (40.6) | — | .91 | ||
|
100,000-150,000 | 344 | 104 (30.2) | — | 318 | 96 (30.2) | — | .99 | ||
|
≥150,000 | 344 | 40 (11.6) | — | 318 | 37 (11.6) | — | .99 | ||
|
||||||||||
|
Depression | 342 | — | 55.37 (6.20) | 315 | — | 55.47 (6.43) | .85 | ||
|
Anxiety | 342 | — | 59.83 (6.95) | 315 | — | 60.00 (7.11) | .75 | ||
Perceived Stress Scale | 342 | — | 2.89 (0.56) | 315 | — | 2.87 (0.60) | .69 | |||
Distressc (composite) | 342 | — | 0.00 (0.88) | 315 | — | 0.00 (0.91) | .97 | |||
Perseverative Thinking Questionnaire | 342 | — | 29.89 (10.43) | 315 | — | 29.62 (11.29) | .76 | |||
Five Facet Mindfulness Questionnaire—Acting with Awareness Subscale | 342 | — | 24.80 (5.93) | 315 | — | 24.56 (6.12) | .62 | |||
National Institutes of Health Toolbox Loneliness | 342 | — | 2.53 (0.77) | 315 | — | 2.58 (0.77) | .45 | |||
Drexel Defusion Scale | 342 | — | 24.83 (7.89) | 315 | — | 24.50 (8.16) | .60 | |||
|
||||||||||
|
Presence | 342 | — | 26.20 (5.44) | 315 | — | 25.81 (5.46) | .36 | ||
|
Search for meaning | 342 | — | 21.63 (6.61) | 315 | — | 22.09 (6.79) | .38 | ||
World Health Organization well-being | 341 | — | 12.76 (4.71) | 315 | — | 12.47 (4.33) | .42 | |||
Self-Compassion Scale | 342 | — | 2.98 (0.69) | 315 | — | 2.93 (0.70) | .37 |
a
bPROMIS: Patient-Reported Outcomes Information System.
cDistress: composite of PROMIS Depression, PROMIS Anxiety, and Perceived Stress Scale.
dMLQ: Meaning in Life Questionnaire.
Higher baseline levels of distress, depression, and stress predicted better outcomes (ie, greater reductions in distress) in HMP (
Baseline variables retained in elastic net models predicting outcomes for each conditiona.
Predictors | Healthy Minds Program model, coefficient | Control model, coefficient | |
Age (years) | —b | — | |
Gender | — | — | |
Race | — | — | |
Marital status | — | −0.006 | |
Income | — | — | |
|
|||
|
Depression | −0.012 | −0.005 |
|
Anxiety | — | −0.007 |
Perceived Stress Scale | −0.003 | −0.006 | |
Distressd (composite) | −0.011 | −0.008 | |
Perseverative Thinking Questionnaire | — | 0.012 | |
Five Facet Mindfulness Questionnaire—Acting with Awareness Subscale | — | — | |
National Institutes of Health Toolbox Loneliness | — | −0.002 | |
Drexel Defusion Scale | — | −0.011 | |
|
|||
|
Presence | — | −0.008 |
|
Search for meaning | — | — |
World Health Organization well-being | — | — | |
Self-Compassion Scale | — | −0.002 |
aThe larger set of baseline predictors retained in the elastic net regularized regression model applied to the control participants relative to the Healthy Minds Program (HMP) group was because the best-fitting model in the former group had a lower
bVariables that were not retained in the elastic net model.
cPROMIS: Patient-Reported Outcomes Information System.
dDistress: composite of PROMIS Depression, PROMIS Anxiety, and Perceived Stress Scale.
eMLQ: Meaning in Life Questionnaire.
The mean PAI score was −0.07 (SD 0.03; range −0.17 to 0.03), indicating that the model predicted greater average symptom improvement for the HMP meditation app than for the assessment-only control condition. The model recommended HMP (PAI<0) for all participants except 5 (657/662, 99.2%).
A significant group × PAI interaction emerged in predicting outcome (
Group × Personalized Advantage Index (PAI) interaction. As PAI scores decrease (ie, reflecting relatively stronger recommendations for the Healthy Minds Program [HMP] app) group differences in observed outcome increase, favoring HMP.
In the linear regression comparison model applied to the HMP group, higher levels of repetitive negative thinking were significantly associated with a
The correlation between predicted control condition outcomes and observed outcomes was
A significant group × PAI interaction emerged in predicting changes in distress (
Group × Personalized Advantage Index (PAI) interaction for the comparison model (ie, linear regression with baseline repetitive negative thinking [PTQ] scores as the sole predictor). As PAI scores decrease (ie, reflecting relatively stronger recommendations for the Healthy Minds Program [HMP] app) group differences in observed outcome increase, favoring HMP.
To demonstrate the translation of a predictive model to personalized recommendations, we used the parameter estimates from the above regression models to estimate predicted changes in distress in HMP versus the assessment-only condition for a new individual based on their preintervention repetitive negative thinking score. Given that the simpler regression model performed similarly to the more complex multivariable ENR models, we used the former model for this demonstration.
First, as shown in
To illustrate with a concrete example, an individual with a repetitive negative thinking (PTQ) score of 1 SD above the mean (ie, 41) would have a PAI score of −0.10 (within the “green zone” of
Plot of the relationship between Personalized Advantage Index (PAI) scores and outcome for each condition to inform personalized recommendations. The dashed vertical gray line indicates the point at which the 2 regression lines intersect (left margin of a bootstrapped 95% CI is shown with a dashed vertical red line). The solid vertical gray line (adjacent to the red line) is derived from the Johnson-Neyman technique and represents the value of the moderator (PAI) at which between-group differences in outcome become statistically significant. Refer to the detailed description in text, with an example for personalized Healthy Minds Program [HMP] recommendation.
An increasing number of individuals are turning to meditation apps to alleviate their emotional distress. Meditation apps represent the most commonly used mental health apps for depression and anxiety [
Critically, a comparison linear regression model that only included information about baseline levels of repetitive negative thinking performed comparably well to a multivariable machine learning model (in contrast, refer to the studies by Webb et al [
As illustrated in
Although potentially useful in terms of encouraging the optimal use of users’ time and attention, informing some individuals that engagement with a meditation app may not be beneficial to them is unlikely to be embraced by many intervention developers. However, these models can be readily extended to instances in which one or more mHealth interventions are being compared. Given the thousands of available mental health apps [
In addition to informing consumer choice, the ability to predict who is most likely to benefit from a particular intervention could inform health care policy and decision-making. In contrast to a stepped care model in which treatment intensity is escalated based on the response to interventions, predictive models could be used to initially assign patients to the treatment expected to yield the best outcomes for that individual based on their baseline characteristics (ie, stratified care) [
Another important avenue for future research is to test the extent to which these findings can be generalized to other meditation apps (eg, Headspace and Calm). In many ways, HMP is similar to other meditation apps. It includes training in mindfulness and connection (eg, loving-kindness, compassion) practices that are also available in popular mindfulness apps such as Headspace and Calm. One difference is that HMP includes practices designed specifically to cultivate a healthy sense of self (Insight module) as well as purposes and meaning in life (Purpose module). The inclusion of these practices is derived from a neuroscience-based model of well-being on which HMP is based [
Finally, given the lack of prior research predicting mental health app outcomes, further research is needed to test the impact of presenting predicted mindfulness app prognosis on patient outcomes. For example, before using a mindfulness app, patients could be randomly assigned to receive their predicted outcomes or not receive this information. Several relevant outcomes could be examined, including (1) between-group differences in symptom change, (2) the extent to which receiving these predictions influences expectancies of therapeutic benefit, (3) the relationship between expectancies and app outcome, and (4) the extent to which individuals use the algorithm-recommended intervention or disregard the recommendation.
This study has several important limitations. First, although basing models exclusively on self-reported data is attractive from an implementation perspective, we may have excluded other patient characteristics that provide important additional predictive information to inform optimal treatment recommendations (eg, biomarkers and cognitive tasks) [
This study demonstrated the potential utility of data-driven approaches for informing personalized meditation app recommendations. A natural extension of this study is to conduct a prospective test of our algorithm using a doubly randomized design. For example, participants could be randomized to either (1) random treatment assignment (ie, treatment A or treatment B) or (2) be assigned to their algorithm-indicated treatment. To the extent that patient outcomes are significantly (and clinically meaningfully) better in the latter condition, the results would support the clinical benefits of algorithm-informed treatment recommendations (for a recent example of a similar design testing predictive matching of patients to therapists, refer to the study by Constantino et al [
Other potentially fruitful future directions include evaluating a broader set of patient characteristics previously shown or hypothesized to predict the likelihood of responding to different interventions [
Supplement.
CONSORT e-HEALTH checklist.
cognitive behavioral therapy
Consolidated Standards of Reporting Trials
cross-validation
Healthy Minds Program
mobile health
Meaning in Life Questionnaire
Personalized Advantage Index
Patient-Reported Outcomes Information System
Perseverative Thinking Questionnaire
randomized controlled trial
Research Electronic Data Capture
root mean square error
This research was supported by National Center for Complementary and Integrative Health grant K23AT010879 (SBG), National Institute for Mental Health grant R01MH43454 (RJD), Chan Zuckerberg Initiative Grant 2020-218037 (RJD), a National Academy of Education/Spencer Postdoctoral Fellowship (MJH), and funding from the Wisconsin Center for Education Research (SBG). The first author (CAW) was partially supported by National Institute for Mental Health R01MH116969, National Center for Complementary and Integrative Health R01AT011002, the Tommy Fuss Fund, and a NARSAD Young Investigator Grant from the Brain & Behavior Research Foundation. SBG was partially supported by the Hope for Depression Research Foundation, Defeating Depression Award.
RJD is the founder, president, and serves on the board of directors for the nonprofit organization Healthy Minds Innovations, Inc. MJH has been a paid consultant at Healthy Minds Innovations, Inc for work unrelated to this study.