This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Mental health problems are highly prevalent, and there is need for the self-management of (mental) health. Ecological momentary interventions (EMIs) can be used to deliver interventions in the daily life of individuals using mobile devices.
The aim of this study was to systematically assess and meta-analyze the effect of EMI on 3 highly prevalent mental health outcomes (anxiety, depression, and perceived stress) and positive psychological outcomes (eg, acceptance).
PsycINFO and Web of Science were searched for relevant publications, and the last search was done in September 2015. Three concepts were used to find publications: (1) mental health, (2) mobile phones, and (3) interventions. A total of 33 studies (using either a within- or between-subject design) including 43 samples that received an EMI were identified (n=1301), and relevant study characteristics were coded using a standardized form. Quality assessment was done with the Cochrane Collaboration tool.
Most of the EMIs focused on a clinical sample, used an active intervention (that offered exercises), and in over half of the studies, additional support by a mental health professional (MHP) was given. The EMI lasted on average 7.48 weeks (SD=6.46), with 2.80 training episodes per day (SD=2.12) and 108.25 total training episodes (SD=123.00). Overall, 27 studies were included in the meta-analysis, and after removing 6 outliers, a medium effect was found on mental health in the within-subject analyses (n=1008), with
Results showed that there was a small to medium effect of EMIs on mental health and positive psychological well-being and that the effect was not different between outcome types. Moreover, the effect was larger with additional support by an MHP. Future randomized controlled trials are needed to further strengthen the results and to determine potential moderator variables. Overall, EMIs offer great potential for providing easy and cost-effective interventions to improve mental health and increase positive psychological well-being.
One in every 3 individuals worldwide will be affected by one or more mental health problems during their lives [
One method that can be used to enhance health self-management is ecological momentary interventions (EMIs) [
Training people in situ could be highly relevant for learning new, healthy behaviors, considering that people under stress typically switch from
Given that the number of worldwide mobile phone users is immense and continues to expand [
This systematic review and meta-analysis therefore attempts to expand the current knowledge by including both mental health outcomes (ie, perceived stress, anxiety, or depressive symptoms) and positive psychological outcomes (eg, positive affect or acceptance). For this quantitative analysis, randomization and the presence of a control group were optional. Although the absence of randomization and the lack of a control group may weaken the design and thus the ensuing conclusions, these criteria are necessary to ensure that the presented overview of EMI studies is complete. This is considered critical because an extensive overview is currently lacking. It should be noted that study design was used in the moderator analyses.
Considering that the access to care needs improvement and EMIs can be used for this, it is important to investigate for whom these technologies can be appropriate and what EMI characteristics are associated with increased effects. Therefore, potentially promising moderators of effect size were investigated. Specifically, sample, type of training, how the training was triggered (ie, automatically or on-demand), support of mental health professional (MHP), and dosage were included because these can be considered key intervention components [
The preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines were followed [
To find relevant publications concerning EMIs that target mental health, a database search was conducted in both PsycINFO and Web of Science (Core Collection). The search strings that were used consisted of 3 groups of words, namely words related to: (1) mental health, (2) mobile phones, and (3) interventions. See
To ensure that no relevant publications were missed with the aforementioned search strategies, an extra search with a similar search string was conducted in the PubMed database on November 2, 2015. This resulted in 3505 publications, and the first 10% was screened to determine whether potentially relevant studies had been missed. However, no relevant publications—that had not already been identified in the other databases—were found, indicating that the used search strategies were sufficient.
Titles and abstracts of publications were first screened for eligibility, and if insufficient information was described in the abstract, the full-text papers were obtained. When a full-text paper was not available, a request was sent to the authors. A number of inclusion criteria were used for both within- and between-subject studies, which were established by authors AV, BV, and JB. First, publications were included when an EMI was studied (eg, via smartphone or personal digital assistant)—either as a stand-alone intervention or in combination with other treatment components. Second, the EMI should be automated and operated independently from a therapist. Thus, studies were excluded when the therapist administered the therapy—for instance—via mobile phone or conference call. This criterion was chosen because of our interest in how new technologies could be used to deliver
To collect the relevant study characteristics from each publication, a standardized form was used. Using this form, the following data were collected: (1) first author and publication year, (2) design, (3) sample characteristics (clinical characteristics, age, gender, and sample size), (4) outcome type, (5) information on the EMI (training type, training trigger, number of training episodes, and whether training was supported by an MHP), and (6) type of control condition and sample size. When a publication reported on more than 1 EMI, information was extracted separately for each described EMI, and all EMIs were included separately in the within-subject analyses. For the between-subject analyses, however, only 1 EMI was included thereby ensuring that each participant is represented only once in the analyses [
In the meta-analysis, the primary outcome of interest was “mental health.” Mental health encompasses an anxiety, depression, or stress outcome. Per publication, a set of guidelines was used to determine which specific questionnaire was used to represent this primary outcome. If a study reported 1 primary outcome, this measure was chosen as an indicator of mental health. When no or multiple primary outcomes were defined, a measure was chosen that was most likely to be affected given the aim of the training. For example, if the training focused on reducing anxiety, then, an anxiety questionnaire was preferred over a questionnaire measuring depression. In this process of selecting questionnaires, comprehensive questionnaires were chosen over restricted questionnaires (if there was such a choice), and the most valid questionnaire was chosen (idem). In addition to the coding of the primary outcome for each publication, the different outcome types per study were also coded. Thus, all questionnaires measuring anxiety, depression, perceived stress, and positive psychological well-being outcomes were listed per publication. A questionnaire was considered to represent positive psychological well-being, when it specifically identified positive emotions or processes that were targeted with the intervention. The only positive psychological well-being outcomes that were identified in the publications were acceptance, feelings of relaxation and quality of life; positive affect, for instance, was not studied in the included publications. By listing all the questionnaires that measured mental health and positive psychological well-being, it was possible to examine whether the effectiveness of EMI differed per outcome type (eg, anxiety or depression).
With regard to the information on the EMI, it was reported whether the training was active or passive. A training was labeled as active when participants had to carry out an exercise, for instance, a relaxation exercise [
The risk of bias in individual studies was assessed using the Cochrane Collaboration tool [
The quality assessment was done by the first author (AV), and a 20% sample was assessed by a second reviewer (MvdP). Inter-rater reliability, as assessed with Cohen’s kappa, indicated that there was moderate agreement between raters (ie, average kappa of .69).
Hedges’
To estimate the effect of EMI from pre intervention to postintervention, analyses were first run with all within-subject data. Furthermore, to determine whether this effect differed from a control condition, between-subject analyses were run. In both the within- and between-subject analyses, it was determined whether there was an effect on the primary outcome “mental health” (as measured with a single questionnaire). Second, it was investigated whether the effect differed per outcome type. That is, was the effect of EMI different for anxiety, depression, perceived stress, or positive psychological outcomes (acceptance, relaxation, and quality of life). To determine the effectiveness per outcome type, all relevant outcome types per publication were included in the analysis. When a study used multiple questionnaires to assess an outcome type (eg, anxiety), an overall mean was created by combining these different questionnaires. By combining multiple questionnaires per study, the data are unlikely to be independent, and this increases the type II error. Therefore, these analyses are only used to explore whether there are potential differences in effects between the outcome types. In addition, for the primary outcome “mental health,” subgroup analyses are done to determine whether the effect differed as a function of design (randomized controlled trial [RCT] or pre-post), sample (healthy or clinical), age, gender, sample size, training type (active or passive), training trigger (triggered, on-demand, or unspecified), daily training episodes (number), total training episodes (number), support by MHP (stand-alone EMI, MHP-supported EMI, or stand-alone EMI with access to care as usual), and quality assessment (0-6). Year of publication was not included as a moderator because there was little variation in this variable (ie, 25 of the 32 publications were published in 2010 or later). Moreover, type of control condition was not included as a moderator because only 13 studies had a between-subject design.
As a measure of heterogeneity, the
Outliers were identified using the value of the standardized residual in both the within- and between-subject analyses. Studies whose standardized residual was significant (values ± 1.96) were excluded from the analyses.
The software Comprehensive Meta-Analysis version 3.3.070 (Biostat) was used for all the described analyses including the calculation of effect sizes with 95% CIs. The forest plots were made using the metaphor package in R (version 3.0.3) [
A total of 2611 publications were identified with the search strategies after removing duplicates (see
For the meta-analysis, 5 publications were excluded because no means and SDs to calculate the effect size were reported or obtained after contacting the authors [
Characteristics of the ecological momentary intervention studies (part 1).
Studya | Designb | Sample | Age (years) | Gender (% female) | nc | Mental Health Measured | Outcome type(s) | |||||||||
Agyapong et al, 2012e | RCT | Clinical | 48.00 | 54 | 24 | BDI | Depression | |||||||||
Ahtinen et al, 2013 | Prepost | Healthy | — | 60 | 14 | Stress single-item | Stress |
|||||||||
Aikens et al. 2015f (all pooled subjects) | Prepost | Clinical | 51.40 | 79 | 221 | PHQ-8 | Depression | |||||||||
Askins et al, 2009 | RCT | Healthy | 36.30 | 100 | 64 | POMS | Depression | |||||||||
Ben-Zeev et al, 2014 | Prepost | Clinical | 45.90 | 39 | 32 | BDI | Depression | |||||||||
Burns et al, 2011e | Prepost | Clinical | 37.40 | 88 | 7 | GIDS-c | Depression |
|||||||||
Carissoli et al, 2015 | RCT | Healthy | 38.11 | 57 | 20 | MSP | Stress | |||||||||
Dagöö et al. 2014g (mCBT) | RCT | Clinical | 34.70 | 48 | 24 | LSAS-SR | Depression |
|||||||||
Dagöö et al, 2014g (mIPT) | RCT | Clinical | 39.08 | 56 | 19 | LSAS-SR | Depression |
|||||||||
Depp et al, 2015 | RCT | Clinical | 46.90 | 54 | 41 | MADRS | Depression | |||||||||
Enock et al. 2014 | RCT | Clinical | 34.80 | 48 | 120 | SIAS | Depression |
|||||||||
Granholm et al, 2012 | Prepost | Clinical | 48.70 | 31 | 41 | BDI | Depression | |||||||||
Grassi et al, 2007 (Vnar) | Preposth | Healthy | 23.27 | 50 | 30 | STAI-state | Anxiety |
|||||||||
Grassi et al, 2007 (Nnar) | Preposth | Healthy | 23.27 | 50 | 30 | STAI-state | Anxiety |
|||||||||
Grassi et al, 2007e (MP3) | Preposth | Healthy | 23.27 | 50 | 30 | STAI-state | Anxiety |
|||||||||
Harrison et al, 2011 | Prepost | Clinical | 38.20 | 71 | 28 | DASS total score | Depression |
|||||||||
Huffziger et al, 2013i | Prepost | Healthy | 22.90 | 60 | 46 | Valence 2-items | Depression |
|||||||||
Kenardy et al, 2003e | RCT | Clinical | 36.80 | 76 | 41 | Anxiety composite score | Anxiety | |||||||||
Lappalainen et al, 2013 | RCT | Clinical | 47.10 | 0 | 11 | GSI | Depression |
|||||||||
Ly et al, 2014e (behavioral activation) | RCT | Clinical | 36.60 | 70 | 36 | BDI | Depression |
|||||||||
Ly et al, 2014 (mindfulness) | RCT | Clinical | 35.60 | 71 | 36 | BDI | Depression |
|||||||||
Ly et al, 2012 | Prepost | Healthy | 29.50 | 36 | 11 | DASS stress | Depression |
|||||||||
Newman et al, 2014 | RCT | Clinical | 42.45 | 55 | 11 | STAI—trait | Anxiety | |||||||||
Newman et al, 1997 | RCT | Clinical | 38.00 | 83 | 9 | FQ—total score | Anxiety | |||||||||
Pallavicini et al, 2009 (VRMB) | Preposth | Clinical | 41.25 | — | 4 | GAD7 | Anxiety | |||||||||
Pallavicini et al, 2009 (VRM) | Preposth | Clinical | 48.50 | — | 4 | GAD7 | Anxiety | |||||||||
Proudfoot et al, 2013 | RCT | Clinical | 39.00 | 70 | 126 | DASS total score | Depression |
|||||||||
Repetto et al, 2013 (VRMB) | Preposth | Clinical | — | 64 | 7 | BAI | Anxiety | |||||||||
Repetto et al, 2013 (VRM) | Preposth | Clinical | — | 64 | 9 | BAI | Anxiety | |||||||||
Rizvi et al, 2011 | Prepost | Clinical | 33.86 | 82 | 22 | BSI | Depression | |||||||||
Shapiro et al, 2010 | Prepost | Clinical | 26.30 | 100 | 14 | BDI | Depression | |||||||||
Watts et al, 2013e | RCT | Clinical | 41.00 | 80 | 10 | BDI | Depression |
|||||||||
Wenze et al, 2014 | Prepost | Clinical | 40.86 | 71 | 14 | QIDS-c | Depression | |||||||||
Gorini et al, 2010 (VRMB) | Preposth | Clinical | — | — | 8 | BAI | Anxiety | |||||||||
Gorini et al, 2010 (VRM) | Preposth | Clinical | — | — | 4 | BAI | Anxiety | |||||||||
Grassi et al, 2011 (Vnar) | Preposth | Healthy | 20.86 | 100 | 15 | STAI-state | Anxiety |
|||||||||
Grassi et al, 2011 (MP3) | Preposth | Healthy | 20.86 | 100 | 15 | STAI-state | Anxiety |
|||||||||
Preziosa et al, 2009 (Vnar; study 1) | Prepost | Healthy | 23.48 | 100 | 6 | STAI-state | Anxiety |
|||||||||
Preziosa et al, 2009 (MP3; study 1) | Prepost | Healthy | 23.48 | 100 | 6 | STAI-state | Anxiety |
|||||||||
Preziosa et al, 2009 (study 2) | RCT | Healthy | 23.48 | 50 | 30 | STAI-state | Anxiety |
|||||||||
Riva et al, 2006 | RCT | Healthy | 23.82 | 48 | 11 | STAI-state | Anxiety |
|||||||||
Zautra et al, 2012 (mindfulness) | RCT | Clinical | 54.05 | 82 | 25 | Depression 3-items | Depression |
|||||||||
Zautra et al, 2012 (mastery-control) | RCT | Clinical | 54.05 | 82 | 25 | Depression 3-items | Depression |
aStudies are ordered by inclusion in the meta-analysis. Behind the study’s year of publication, between brackets, the sample (or condition) that received the ecological momentary intervention was specified; With mCBT: mobile cognitive behavioral therapy; mIPT: mobile interpersonal psychotherapy; MP3: audio only condition; Nnar: video only condition VRMB: virtual reality and mobile condition with biofeedback; VRM: virtual reality with mobile condition; Vnar: video narrative condition.
bDesign of study is labeled either randomized controlled trial (RCT) or prepost design.
cSample size at post-intervention in the condition receiving the ecological momentary intervention.
d The specific questionnaire that was used to represent the primary outcome “mental health” is listed. With BDI: Beck Depression Inventory; PHQ-8: Personal Health Questionnaire Depression scale; POMS: Profile of Mood States; GIDS-c: Quick Inventory of Depressive Symptoms-Clinician rated; MSP: Mesure du Stress Psychologique; LSAS-SR: Liebowitz Social Anxiety Scale Self-Report; MADRS: Montgomery–Åsberg Depression Rating Scale; SIAS: Social Interaction Anxiety Scale; BAI: Beck Anxiety Inventory; STAI: State-Trait Anxiety Inventory; DASS: Depression Anxiety Stress Scales; GSI: General Symptom Index; FQ: Fear Questionnaire; GAD7: Generalized Anxiety Disorder 7-item; BSI: Brief Symptom Inventory.
eStudy is considered an outlier in within-subject analyses.
fThe data used for the analyses consist of all pooled participants, the outcome questionnaire at pre-intervention is compared with last outcome questionnaire that participant completed.
gThe intervention could be accessed using the mobile phone, tablet, and computer.
hStudy is labeled as a prepost design because it is unclear whether participants were randomized across conditions.
iThe study technically is an ecological momentary assessment study with an experimental manipulation.
Characteristics of the ecological momentary intervention studies (part 2).
Studya | Intervention technique | Training type (+ type of MHPb supportc) | Training trigger | No. of training sessionsd | Control (n)e | ||||
Agyapong et al, 2012f | Self-management and monitoring | Passive (stand-alone + CAU) | Triggered | 168 (2) | Waitlist (n=28) | ||||
Ahtinen et al, 2013 | Acceptance and commitment therapy | Active | On-demand | ||||||
Aikens et al, 2015g |
Self-management and monitoring | Passive (+MHP) | Triggered | 26 (1) | |||||
Askins et al, 2009 | Self-management and monitoring | Active (+MHP) | ... | ... | |||||
Ben-Zeev et al, 2014 | Self-management and monitoring | Active (+stand-alone + CAU) | Triggered | 90 (3) | |||||
Burns et al, 2011f | Behavioral activation | Active (+MHP) | Triggered | 280 (5) | |||||
Carissoli et al, 2015 | Mindfulness | Active | On-demand | 36 (2) | Placebo (n=18) | ||||
Dagöö et al, 2014h |
Cognitive behavioral therapy | Active (+MHP) | ... | ... | |||||
Dagöö et al 2014h |
Interpersonal therapy | Active (+MHP) | ... | ... | |||||
Depp et al, 2015 | Self-management and monitoring | Passive (+MHP) | Triggered | 140 (2) | Paper and pencil version (n=41) | ||||
Enock et al, 2014 | Cognitive bias modification | Active | Triggered | 84 (3) | Placebo (n=104) | ||||
Granholm et al, 2012 | Cognitive behavioral therapy | Active (stand-alone + CAU) | Triggered | 216 (3) | |||||
Grassi et al, 2007 (Vnarb) | Relaxation | Active | ... | 4 (2) | Waitlist |
||||
Grassi et al, 2007 (Nnarb) | Relaxation | Active | ... | 4 (2) | |||||
Grassi et al, 2007f (MP3b) | Relaxation | Active | ... | 4 (2) | |||||
Harrison et al, 2011 | Self-management and monitoring | Passive | On-demand | ... | |||||
Huffziger et al, 2013i | Mindfulness | Passive | Triggered | 10 (10) | |||||
Kenardy et al, 2003f | Cognitive behavioral therapy | Active (+MHP) | Triggered | 420 (5) | CBT6 (n=44) | ||||
Lappalainen et al, 2013 | Cognitive behavioral therapy and acceptance and commitment therapy | Active (+MHP) | On-demand | ... | Waitlist (n=12) | ||||
Ly et al, 2014f |
Behavioral activation | Active (+MHP) | ... | ... | |||||
Ly et al, 2014 mindfulness | Mindfulness | Active (+MHP) | ... | ... | |||||
Ly et al, 2014 mindfulness | Acceptance and commitment therapy | Active | On-demand | ... | |||||
Newman et al, 2014 | Cognitive behavioral therapy | Active (+MHP) | Triggered | 112 (4) | CBT6 (n=14) | ||||
Newman et al, 1997 | Cognitive behavioral therapy | Active (+MHP) | Triggered | 336 (4) | CBT12 (n=9) | ||||
Pallavicini et al, 2009 |
Relaxation | Active (+MHP) | On-demand | ... | Waitlist (n=4) | ||||
Pallavicini et al, 2009 |
Relaxation | Active (+MHP) | On-demand | ... | |||||
Proudfoot et al, 2013 | Self-management and monitoring | Passive | On-demand | ... | Placebo (n=195) | ||||
Repetto et al, 2013 (VRMB) | Relaxation | Active (+MHP) | On-demand | ... | Waitlist (n=8) | ||||
Repetto et al, 2013 (VRM) | Relaxation | Active (+MHP) | On-demand | ... | |||||
Rizvi et al, 2011 | Dialectical behavior therapy | Active (+TAU) | On-demand | ... | |||||
Shapiro et al, 2010 | Self-management and monitoring | Passive (+MHP) | — | 168 (1) | |||||
Watts et al, 2013f | Cognitive behavioral therapy | Active (+MHP) | On-demand | ... | Computer version (n=15) | ||||
Wenze et al, 2014 | Cognitive behavioral therapy | Passive (stand-alone + CAU | Triggered | 28 (2) | |||||
Gorini et al, 2010 (VRMB) | Relaxation | Active (+MHP) | On-demand | ... | Waitlist (n=8) | ||||
Gorini et al, 2010 (VRM) | Relaxation | Active (+MHP) | On-demand | ... | |||||
Grassi et al, 2011 (Vnar) | Relaxation | Active | ... | 6 (1) | Waitlist (n=15) | ||||
Grassi et al, 2011 (MP3b) | Relaxation | Active | ... | 6 (1) | |||||
Preziosa et al, 2009 (Vnar; study 1) | Relaxation | Active | ... | 6 (1) | Waitlist (n=6) | ||||
Preziosa et al, 2009 (MP3; study 1) | Relaxation | Active | ... | 6 (1) | |||||
Riva et al, 2006 | Relaxation | Active | ... | 4 (2) | Placebo (n=30) | ||||
Preziosa et al, 2009 (study 2) | Relaxation | Active | ... | 4 (2) | Placebo (n=11) | ||||
Zautra et al, 2012 (mindfulness) | Mindfulness | Active | Triggered | 27 (1) | Placebo (n=23) | ||||
Zautra et al, 2012 |
Behavioral activation | Active | Triggered | 27 (1) |
aStudies are ordered by inclusion in the meta-analysis. Behind the study’s year of publication, between brackets, the sample (or condition) that received the EMI was specified.
bmCBT: mobile cognitive behavioral therapy; mIPT: mobile interpersonal psychotherapy; MP3: audio only condition; MHP: mental health professional; Nnar: video only condition; Vnar: video narrative condition; VRMB: virtual reality and mobile condition with biofeedback; VRM: virtual reality with mobile condition.
cFollowing the type of training, the type of support by the mental health professional is reported between brackets. With +MHP=mental health professional–supported EMI; stand-alone + CAU=stand-alone EMI with access to care as usual. No information was displayed when the EMI was stand-alone.
dThe maximum number of total training sessions is reported. The maximum number of daily training sessions is reported between brackets.
eControl condition (and sample size at post-intervention) is listed if the study was included in the between-subject analyses. If the control condition is an active treatment, it is specified which specific active treatment condition is used to calculate the effect size. With CBT6=6-sessions of cognitive behavioral therapy; CBT12=12-sessions of cognitive behavioral therapy.
f Study is considered an outlier in within-subject analyses.
gThe data used for the analyses consist of all pooled participants, the outcome questionnaire at preintervention is compared with last outcome questionnaire that participant completed.
hThe intervention could be accessed using the mobile phone, tablet, and computer.
iThe study is technically an ecological momentary assessment study with an experimental manipulation.
PRISMA flow diagram for study inclusion.
Of the 33 studies that were included, 17 had a prepost design, and 16 studies were an RCT. Of the total number of studies, 10 included healthy individuals [
A range of different intervention techniques were studied: CBT [
On average, the EMI lasted for 7.47 weeks (SD=6.46), but this varied considerably. For example, the studies with the shortest EMI lasted only 1 or 2 days [
The training episodes were automatically triggered by the device in 13 studies, and in 11 studies, the training episodes were not specifically triggered, and participants could complete the training whenever they wanted. Nine studies did not report whether the training was triggered or whether it was accessed on-demand.
The quality assessment of the studies is summarized in
Quality assessment of the individual studies using the Cochrane Collaboration’s tool.
Study | Random sequence generationa | Allocation concealmenta | Performance biasb | Detection bias | Attrition biasc | Reporting biasd | Overall gradee | |
Agyapong et al, 2012 | + | − | − | − | + | + | 3 | |
Ahtinen et al, 2013 | N/A | N/A | − | − | + | + | 4 | |
Aikens et al, 2015 | − | − | − | − | + | + | 2 | |
Askins et al, 2009 | + | ? | − | − | − | + | 2 | |
Ben-Zeev et al, 2014 | N/A | N/A | − | − | + | + | 4 | |
Burns et al, 2011 | N/A | N/A | − | ? | + | + | 4 | |
Carissoli et al, 2015 | ? | ? | − | − | + | + | 2 | |
Dagöö et al, 2014 | + | + | − | − | + | + | 4 | |
Depp et al, 2015 | + | + | + | − | − | + | 4 | |
Enock et al, 2014 | ? | ? | ? | − | − | + | 1 | |
Gorini et al, 2010f | ? | ? | − | − | ? | − | 0 | |
Granholm et al, 2012 | N/A | N/A | − | − | − | + | 3 | |
Grassi et al, 2011f | ? | ? | − | − | ? | − | 0 | |
Grassi et al, 2007 | ? | ? | − | − | ? | − | 0 | |
Harrison et al, 2011 | N/A | N/A | − | − | − | + | 3 | |
Huffziger et al, 2013 | + | ? | − | − | + | + | 3 | |
Kenardy et al, 2003 | ? | ? | − | − | ? | + | 1 | |
Lappalainen et al, 2013 | ? | ? | − | − | + | + | 2 | |
Ly et al, 2014 | + | + | − | − | + | + | 4 | |
Ly et al, 2012 | N/A | N/A | − | − | + | + | 4 | |
Newman et al, 2014 | ? | ? | − | − | + | + | 2 | |
Newman et al, 1997 | ? | ? | − | − | + | + | 2 | |
Pallavicini et al, 2009 | + | ? | − | − | + | − | 2 | |
Preziosa et al, 2009f (studies 1 and 2) | ? | ? | − | − | ? | − | 0 | |
Proudfoot et al, 2013 | + | + | + | − | − | + | 4 | |
Repetto et al, 2013 | + | ? | − | − | + | − | 2 | |
Riva et al, 2006f | ? | ? | − | − | ? | − | 0 | |
Rizvi et al, 2011 | N/A | N/A | − | − | + | + | 4 | |
Shapiro et al, 2010 | N/A | N/A | − | − | − | + | 3 | |
Watts et al. 2013 | + | + | − | − | − | + | 3 | |
Wenze et al, 2014 | N/A | N/A | − | ? | + | + | 4 | |
Zautra et al, 2012f | ? | ? | − | − | + | + | 2 | |
aThe label “not applicable” (N/A) is used in 1-armed studies.
bThe risk for performance bias is rated low if personnel are blinded irrespective of whether participants were blinded.
cThe bias for attrition is considered high when the attrition from pre-intervention to post-intervention is 20% or more.
dThe bias for selective reporting is labeled low if all prespecified outcomes are reported, it is not necessary that all statistical information is reported per outcome (eg, means, standard deviation, CI,
eThe overall grade is determined by summing the number of low-risk categories and the number of N/A categories; +=low risk of bias; −=high risk of bias; ?=unclear risk of bias.
fStudy is not included in the meta-analysis.
A total of 27 publications including 33 EMI groups (n=1156), were included in the within-subject analyses, and these studies had significant heterogeneity,
The average effect on mental health from pre-intervention to post-intervention was
The standardized residual identified 6 studies as outliers, and these were removed from the analyses [
It was explored whether the effect was different per outcome type. Depressive symptoms were assessed in 17 studies; anxiety in 15 studies; quality of life in 6 studies; stress in 5 studies; acceptance in 4 studies, and relaxation in 3 studies. As can be seen in
Furthermore, subgroup analyses were done to see whether the effect varied by moderator.
Effect sizes (Hedges’
Outcome | Random effect model | Heterogeneity | Test of difference | ||||||
nc | |||||||||
Mental health | 27 | 1008 | 0.57 (0.45-0.70)g | 74.46g | 65.08 | ||||
Design | 1.03 | ||||||||
RCTh | 11 | 481 | 0.65 (0.48-0.82)g | 24.10i | 58.50 | ||||
Pre-post | 16 | 527 | 0.52 (0.33-0.71)g | 47.34g | 68.32 | ||||
Sample | 1.79 | ||||||||
Clinical | 20 | 793 | 0.63 (0.50-0.76)g | 39.32i | 51.68 | ||||
Healthy | 7 | 215 | 0.40 (0.10-0.71)j | 26.76g | 77.58 | ||||
Agek, years | 2.19 | ||||||||
≤ 38.15 | 12 | 426 | 0.61 (0.36-0.86)g | 54.38g | 79.77 | ||||
> 38.15 | 12 | 552 | 0.51 (0.37-0.64)g | 17.64l | 37.65 | ||||
Unspecified | 3 | 30 | 0.80 (0.41-1.18)g | 0.40 | 0.00 | ||||
Genderk | 1.96 | ||||||||
≤ 60% female | 14 | 450 | 0.49 (0.28-0.70)g | 51.25g | 74.63 | ||||
> 60% female | 11 | 550 | 0.67 (0.53-0.81)g | 15.94 | 37.26 | ||||
Unspecified | 2 | 8 | 0.55 (−0.08 to 1.17)l | 1.12 | 10.43 | ||||
Sample sizek | 1.18 | ||||||||
≤ 22 participants | 13 | 158 | 0.67 (0.46-0.87)g | 17.24 | 30.39 | ||||
> 22 participants | 14 | 850 | 0.52 (0.36-0.69)g | 56.36g | 76.93 | ||||
Training type | 0.32 | ||||||||
Active | 20 | 518 | 0.60 (0.42-0.78)g | 57.51g | 66.96 | ||||
Passive | 7 | 490 | 0.53 (0.34-0.71)g | 16.65j | 63.97 | ||||
Training trigger | 1.65 | ||||||||
Triggered | 9 | 535 | 0.52 (0.33-0.71)g | 26.96i | 70.45 | ||||
On-demand | 11 | 256 | 0.49 (0.37-0.62)g | 9.41 | 0.00 | ||||
Unspecified | 7 | 217 | 0.76 (0.38-1.14)g | 35.69g | 83.19 | ||||
No. of daily training episodesk | 0.53 | ||||||||
≤ 2 | 7 | 370 | 0.55 (0.24-0.87)i | 32.65g | 81.62 | ||||
> 2 | 6 | 259 | 0.51 (0.20-0.82)i | 22.81g | 78.08 | ||||
Unspecified | 14 | 379 | 0.63 (0.49-0.77)g | 17.48 | 25.62 | ||||
No. of total training episodesk | 0.92 | ||||||||
≤ 84 | 7 | 481 | 0.48 (0.21-0.75)i | 36.62g | 83.62 | ||||
> 84 | 6 | 148 | 0.62 (0.27-0.97)i | 17.77i | 71.86 | ||||
Unspecified | 14 | 379 | 0.63 (0.49-0.77)g | 17.48 | 25.62 | ||||
Support MHPm | 6.77j | ||||||||
MHP-supported EMI | 14 | 474 | 0.73 (0.57-0.88)g | 20.67l | 37.10 | ||||
Stand-alone EMI | 9 | 425 | 0.45 (0.22-0.69)g | 35.81j | 77.66 | ||||
Stand-alone EMI with access to care as usual | 4 | 109 | 0.38 (0.11-0.64)i | 5.37 | 43.97 | ||||
Quality assessmentk | 0.01 | ||||||||
≤ 3 | 17 | 781 | 0.57 (0.39-0.76)g | 57.68j | 72.26 | ||||
> 3 | 10 | 227 | 0.59 (0.42-0.76)g | 16.78l | 46.38 |
aOutliers were excluded from the presented moderation analyses (ie, 6 studies).
b
cn=number of participants.
d
e
f
g
hRCT=randomized controlled trial.
i
j
kData were categorized based on the median.
l
mMHP=mental health professional.
Effect sizes (Hedges’
Random effect model | Heterogeneity | Test of difference | ||||||
Outcome | nc | |||||||
Overall | 50 | 1830 | 1.74 | |||||
Anxiety | 15 | 468 | 0.47 (0.32-0.63)g | 28.28h | 50.49 | |||
Depression | 17 | 870 | 0.48 (0.34-0.61)g | 46.48g | 65.58 | |||
Perceived stress | 5 | 199 | 0.40 (0.23-0.57)g | 4.59 | 12.79 | |||
Relaxation | 3 | 106 | 0.28 (−0.46 to 1.01) | 25.28g | 92.09 | |||
Acceptance | 4 | 72 | 0.36 (0.13-0.59)i | 2.79 | 0.00 | |||
Quality of life | 6 | 115 | 0.38 (0.19-0.56)g | 4.25 | 0.00 |
aOutliers were excluded from the presented moderation analyses (ie, 6 studies).
b
cn=number of participants.
d
e
f
g
h
i
Forest plot showing the effect of ecological momentary interventions (EMIs) on mental health complaints for all within-subject studies. The EMI sample (or condition) is reported after the year of publication when multiple EMI samples were included in a publication.
Funnel plot of standard error by Hedges’
In the between-subject analyses, only 1 EMI group per study was included (see “Coding”). A total of 13 studies were included with 454 participants in the EMI condition and 522 participants in a control condition (waitlist, placebo, or active treatment control). The included studies were not significantly heterogeneous,
The effect for EMI in between-subject studies was
Forest plot showing the effect of ecological momentary interventions (EMIs) on mental health complaints for all between-subject studies. The EMI sample (or condition) that was used to represent the active treatment condition is reported after the year of publication.
Funnel plot of standard error by Hedges’
The systematic review and meta-analysis was a first attempt to examine whether mobile technologies can be used to provide an effective intervention for mental health and under which circumstances this is the case. A total of 33 studies (
In the within-subject studies (n=1008), a significant medium effect size (Hedges’
With regard to the between-subject studies (n=454), the estimated effect size was 0.40. The effect was, however, subject to publication bias, and the corrected effect was considered small, but significant (
Both the within- and the between-subject analyses indicate that mobile technologies can be effectively used to deliver interventions for mental health. When interpreting this effect, it must be acknowledged that the effects were considerable smaller in the between-subject studies compared with the within-subject studies. A larger effect in within-subject studies is frequently observed. However, within-subject studies are limited because causality can—generally—not be interfered from these studies. Moreover, these studies have an increased risk for type-II errors, which implies that the conclusions from within-subject studies must be interpreted with caution [
The finding that the effect of EMIs was stronger when support by an MHP was included is in line with findings from research on Internet interventions (eg, [
Apart from the moderator “support by an MHP,” no moderation effects were found for the other study or intervention characteristics. The intervention was, for example, equally effective for healthy versus clinical individuals. The absence of significant moderator variables implies that any form of EMI, irrespective of for instance type of training or number of training episodes, is equally effective for all individuals. Obviously, this assumption is implausible, and it is more likely that the null findings are the result of the relative small number of studies that specifically reported the intervention characteristics (eg, number of training episodes and whether training was triggered) [
This meta-analysis is limited by the low reported study quality (ie, 2.29 on a scale from 0 to 6). When the reported study quality is low, the study may be subject to weakness in the experimental setup or to problems in the processing of the data. These shortcomings can influence the true effect and lead to an overrepresentation or underrepresentation [
In line with the previous limitation, it is also important that sufficient intervention details are described so that other researchers can fully comprehend what the intervention entailed. In the included studies, the content of the intervention was described, yet other important intervention components—as suggested by Davidson et al [
Another limitation is that the larger part of the included studies used a within-subject design. Although this design can yield valuable information, RCTs (which use a between-subject design) are considered superior when evaluating interventions because these can be used to establish a causal relation. Moreover, some of the included studies (both within- and between-subject) had small sample sizes. Studies with small sample sizes may be statistically underpowered to detect an effect and have a lower study validity [
To conclude, the meta-analysis found a small to medium effect of EMIs on mental health, and this effect did not differ across the different outcome types. Furthermore, the effect appeared to be larger when the EMI was supported by an MHP. It is important that future research determines how support by an MHP can best be implemented and if this support is a necessity for everyone. In addition, new research studies should investigate what the active features of an EMI are. Overall, the use of EMIs for improving mental health is supported; EMIs offer great potential for providing easy and cost-effective strategies to improve mental health and positive psychological well-being in the population.
Specific search strings used to find publications.
cognitive behavioral therapy
ecological momentary intervention
mental health professional
not applicable
Preferred Reporting Items Systematic reviews and Meta-Analyses
randomized controlled trial
This work was supported by the “Top”-grant of the Netherlands Organisation for Health Research and Development (ZON-MW) to Jos F. Brosschot, under grant number 40-0081 2-98-1 I 029.
None declared.