A systematic search was conducted using OvidSP to search 7 electronic databases, MEDLINE, Embase, Cochrane Central Register of Controlled Trials, Allied and Complementary Medicine, Health Management Information Consortium, Health Technology Assessment, and PsycINFO, for articles published through May 1, 2020 (Table 1). The search was conducted on May 7, 2020. In brief, the search strategy combined synonyms for the population of interest (patients with mental illness), the intervention modality (mobile phone apps), and the type of study (RCT). Search results were limited to the English language and studies of humans.

Only articles published in peer-reviewed journals were included. Articles were deemed eligible if they were RCTs of mobile app–based interventions targeting adults (aged >18) with clinical depression, in which depressive symptoms were a primary outcome of interest, and retention in posttreatment assessments was reported. We defined a mobile app–based intervention as one that required use of a mobile device app as part of the treatment.

We defined studying a “clinically depressed” sample as meeting at least one of the following criteria: (1) eligibility criteria requiring participants to have scores on a depression self-report measure over an established clinical cutoff; (2) eligibility criteria requiring participants to have a psychiatric diagnosis per their medical record or per a structured clinical interview; or (3) reported average baseline scores on a depression self-report measure above an established clinical cutoff in all groups. When there was ambiguity on the established clinical cutoff for a self-report measure, we used the lowest published cutoff score.

At least two independent reviewers judged article eligibility (JML, JGL, or RVB), with any disagreements resolved through mediation with a third reviewer (TPH). The screening process began with title and abstract review followed by a full-text review of any articles that appeared potentially relevant based on the abstract/title review or where there was insufficient information in the abstract to determine eligibility.

Data extraction occurred in 3 parts. First, data were extracted by one author (JGL or RVB). Next, the rationale for each datapoint and where it came from in the original articles were reviewed with JML. Finally, all datapoints considered ambiguous or disagreements between the authors who completed the initial data extraction and JML were reviewed with one additional author (TPH).

Two kinds of information were extracted from each article. First, intervention details were extracted, including the (1) clinical population, (2) length of the treatment period, (3) a description of the study conditions, (4) total sample size in each condition, and (5) whether human support by a coach or licensed clinician was offered as part of the intervention.

Second, a 5-element framework of minimum necessary DMHI engagement reporting, developed by our study team, was used to extract key descriptive and numeric indices of participant engagement with the intervention. Elements in this framework were as follows: (1) recommended intervention app use as communicated to participants at enrollment and, when reported, intervention app adherence criteria; (2) rate of uptake, defined as the number and percentage of participants randomized to the intervention who engaged with their assigned app at all; (3) level of intervention app use metrics, specifically number of times participants used the app and amount of time participants spent in the app; (4) duration of intervention app use metrics (ie, whether weekly use patterns were reported and the number and percentage of participants who used the app in the final week of the intervention period); and (5) number and percentage of participants randomized who could be considered intervention completers. Furthermore, for context, we identified whether studies used backend data or other methods (such as self-report) to quantify app usage and extracted any additional data presented on intervention engagement.

The full systematic search retrieved a total of 3137 records (Figure 1). Following the removal of duplicate articles across electronic databases, 2083 articles were screened at the title-and-abstract phase. This identified 150 articles as potentially eligible, which were subsequently screened in full. Full-text screening resulted in the exclusion of 128 articles for reasons specified in Figure 1. A total of 22 independent studies [13,15,21-40] were ultimately eligible for inclusion.

Detailed study characteristics are presented in Table 2. While all 22 studies included a clinically depressed sample and symptoms of depression as a primary outcome, the target populations differed. Of the 22 eligible studies, the following target populations were recruited: depression (n=13), suicidal ideation (n=1); depression or anxiety (n=3); bipolar disorder (n=1); medical population with clinically significant symptoms of depression (n=2); community sample (n=1); and college students (n=1). Intervention periods ranged from 2 weeks to 6 months and sample sizes ranged from 30 to 720. Interventions evaluated included a range of human support: 11 were entirely self-help interventions involving no human support, 9 involved a licensed clinician, 1 involved a clinical coach, and 1 included clinical support from research staff for whom licensure status was not specified. For descriptive purposes, apps studied were assigned to 1 of 3 categories: those intended to be used as daily self-management/skill-building tools (n=13); those intended to provide support in the context of clinician-administered care or to facilitate communication with clinicians (n=5); and treatments involving a discrete number of lessons/modules typically to be completed on a weekly basis (n=4).

Data extracted based on our 5-element framework are presented in Table 3 (with additional details presented in Multimedia Appendix 2). With the exception of Ludtke et al [29], all studies that reported on app usage indicated using backend data from the app to monitor app usage in the test condition(s). Ludtke et al [29] only offered self-reported app usage data; 14/22 papers (64%) reported the rate of app uptake defined as the number of participants randomized to the intervention condition(s) who engaged with the app at least once. Findings in those studies reporting the rate of app uptake indicated that between 42% and 100% of those participants randomized to an app-based DMHI condition engaged with the app at least once.

With regard to ongoing use, reports were varied. A total of 13 papers (59%) reported a level-of-use metric. The most common level-of-used metric was number of sessions/launches (n=12). Time spent in the app was a less popular level-of-use metric (n=4). Fewer papers reported metrics on duration of use. Only 5 studies (23%) reported weekly use patterns over the course of the intervention and the number of participants who were still using the intervention during the last week of the treatment period.

With regard to questions of whether participants completed the intervention as intended, reporting was also varied. Table 3 describes the app intervention instructions given to participants and app adherence criteria to the extent that these were specified in each article. Only 3 studies clearly reported the number of participants randomized to the DMHI who were considered to have completed the app-based components of the intervention as intended per specified intervention instructions. An additional 4 studies (footnote i in Table 3) reported the number of participants who met a specified adherence threshold such as using the intervention app once per week; 4 more studies reported metrics related to intervention completion, including percentage of patients who used the app on a daily basis (n=1; footnote m); percentage of patients completing the intervention based on a criterion defined by clinician contact rather than app use (n=2; marked by footnote o); and percentage of participants who downloaded all the intervention content (n=1; marked by footnote t). Findings in those studies reporting metrics related to intervention completion indicated that between 14.4% and 93.0% of participants randomized to a DMHI condition completed the intervention as intended or according to a specified app adherence criteria. Among the 11 studies reporting this metric, 6 reported that less than or equal to 50% of participants completed the intervention.

This scoping review has revealed that reporting on engagement with DMHIs in RCTs is highly variable. A number of basic metrics of intervention engagement, such as rate of intervention uptake, weekly use patterns, and number of intervention completers, were routinely not reported. When intervention engagement metrics were reported, it was common to see low levels of engagement. The variability in reporting and frequency of low engagement when reported highlight the importance of establishing minimum necessary reporting standards for engagement in DHMI research.

Only 64% (14/22) of studies included in this review specified rate of uptake, defined as the number of participants randomized to the intervention condition who used the app at least once. Past research suggests that rate of uptake cannot be assumed, especially in the context of fully remote, self-guided digital interventions. Those studies that did report this metric showed varied levels of uptake. For example, Arean at al [21] found that over one-half of participants did not download their assigned app, whereas Roepke et al [38] and Watts et al [13] found that closer to one-quarter of participants did not download their assigned app. The studies reviewed here varied in the type of app and design so different rates of uptake may be expected, but the extent of inconsistent reporting was surprising.

Level of use metrics, defined as both the number of app launches and the amount of time the intervention was used, was only reported in 59% (13/22) of the studies reviewed. These metrics—specifically, average number of uses and average time spent in the app—should be feasible to calculate when researchers have access to activity log data of the tested app, which was the case in most of the studies included. There can be some complications reporting these metrics. For example, it can be difficult to accurately report time spent in the app when participants leave an app open on their device for longer than they are actively using it. Similarly, apps can be launched only to be closed in a matter of seconds. However, in cases where these metrics are not appropriate for the intervention being evaluated, we would have expected to see alternative metrics such as number of clicks reported, but this was only the case in 1 of the reviewed studies [36].

Approximately one-quarter of studies (5/22, 23%) reported on participant duration of use, defined as reporting both weekly use patterns and the number of participants who used the app in the final week of the intervention period. It is well documented that, in general, mobile apps tend to be used heavily when first downloaded and that use decreases over time [41]. Similarly, concerns related to sustained engagement with web-based psychiatric interventions have been reported in routine-care implementation studies [4-6,17]. Inconsistent use of psychiatric intervention apps over time is an issue that needs to be addressed if our field is to mature; however, addressing this issue will be all the more difficult if such variations in use are not adequately reported in our published literature. Data from Dahne et al [25] provide an excellent example of how this metric is useful to report alongside level of use. They reported that 81.8% of participants in the intervention condition used the app at least eight times (an average of at least once per week), but only 51% of participants used the app during the last week. Much like patterns of use with other popular apps, these data suggest high initial use that declines over time.

In the context of intervention research, it is important to include some clear metric of intervention adherence or completion. Yet only 50% (11/22) of studies in this review clearly reported the number of participants considered to have completed the app-based components of the intervention as intended or other metrics related to completion such as percentage of patients who met a specified adherence threshold; percentage of patients completing clinician-based components of the intervention; and percentage of participants who downloaded all the intervention content. Just like psychotherapy or medication use, mobile app–based interventions incorporate some expected efficacious dose into the instructions for use. The fact that use can be accurately and objectively tracked from backend metrics is highly encouraging, and distinguishes our field from other treatment research (such as medication trials) where adherence has historically been extremely difficult to reliably measure. Further, completion need not be full use exactly as intended. For example, Arean et al [21] specified that 50% compliance with intervention instructions was considered completion. Simply not discussing who uses mobile app depression interventions as intended, however, will limit the potential for insight into and utility of these interventions.

Finally, one of our objectives in this review was to quantify standard level of engagement in RCTs of mobile app–based depression interventions. Our data extraction led us to conclude that with the current state of reporting, this is nearly impossible to do. What we did conclude is that engagement at all points—uptake, level, duration, and completion—is widely varied. Moreover, it was not uncommon to see completion rates at or below 50% of those participants randomized to a treatment condition (n=6) or to simply see engagement rates not reported at all (n=5).

This scoping review has several limitations. First, this review illustrates an important dilemma in the field of DMHI research, but findings are limited to a subset of DMHI literature, specifically only that involving depression interventions in psychiatric samples with mobile app–based interventions. While we expect our proposed reporting guidelines to be useful across DMHIs, the extent to which the findings of this review carry through to mobile app interventions in other areas of mental health remains unclear. Second, our original goal in approaching this scoping review was to quantify typical engagement with DMHIs in RCTs; however, as we began the literature review, we ascertained that this goal would be difficult given the variability (and often absence) of metrics reported. This study, therefore, represents a shift in objectives. Third, we only reviewed papers from academic sources, which limits the kinds of mental health apps we took into account. The quality and objectivity of the data contained within independently published reports from private industries on their own mental health apps have yet to be reviewed. Finally, this review only evaluated literature though May 2020. While there is no reason to expect that reporting on engagement has improved, this work should be conceptualized as only a starting point for a discussion of appropriate reporting guidelines and future reviews or meta-analyses on this topic are warranted.

The emerging field of DMHIs has reached a critical juncture: intervention engagement has been widely recognized as the key factor limiting DMHI clinical utility. This review illustrates that engagement is variable and frequently underreported. Adopting a set of reporting guidelines that specify the minimum necessary information when publishing RCTs of DMHIs will provide new insights into how to improve engagement in mental health apps; allow for clear comparisons between DMHIs and other treatment options; and offer benchmarks upon which further research must improve. Such reporting standards will complement the expanding literature on user-centered evaluations of engaging with digital health tools and interventions [42-44].

To this end, we suggest the 5-element framework applied in this study be used to guide minimum necessary DMHI engagement reporting standards. This framework includes the following: (1) intervention instructions or adherence criteria, defined as an explicit statement of what it means for participants to have used an intervention as intended or met some minimum intervention threshold; (2) rate of uptake, defined as the number of participants randomized to the intervention who downloaded the associated app(s) and used them at least once; (3) level of use metrics, defined as both the number of app launches and the amount of time the intervention was used (with alternative metrics such as number of clicks appropriate if more suitable for the intervention and justified); (4) duration of use, defined as participants’ weekly use patterns; and (5) number of completers, defined as the number of participants who completed the intervention as intended per intervention instructions or per specified adherence criteria. We believe this framework could be a useful starting point to promote standards of reporting within the field, with room for future iterations.

Certainly complexities exist when identifying and reporting engagement with DMHIs given that these interventions vary widely in content and format. The reporting guidelines that we have suggested in response to our findings are intended both to be broadly applicable across DMHIs and to challenge the field to move past complexities and move toward greater transparency and rigor. We hope this begins an important discussion on reporting standards that will improve our understanding of how to evaluate and optimize DMHIs.