Published on in Vol 21, No 10 (2019): October

Methodological Challenges in Randomized Controlled Trials on Smartphone-Based Treatment in Psychiatry: Systematic Review

Methodological Challenges in Randomized Controlled Trials on Smartphone-Based Treatment in Psychiatry: Systematic Review

Methodological Challenges in Randomized Controlled Trials on Smartphone-Based Treatment in Psychiatry: Systematic Review


1Copenhagen Affective Disorder Research Center, Psychiatric Center Copenhagen, Rigshospitalet, University Hospital of Copenhagen, Copenhagen, Denmark

2Department of Health Technology, Technical University of Denmark, Lyngby, Denmark

Corresponding Author:

Morten Lindbjerg Tønning, MD

Copenhagen Affective Disorder Research Center

Psychiatric Center Copenhagen, Rigshospitalet

University Hospital of Copenhagen

Blegdamsvej 9

Copenhagen, 2100


Phone: 45 28975755


Background: Smartphone-based technology is developing at high speed, and many apps offer potential new ways of monitoring and treating a range of psychiatric disorders and symptoms. However, the effects of most available apps have not been scientifically investigated. Within medicine, randomized controlled trials (RCTs) are the standard method for providing the evidence of effects. However, their rigidity and long time frame may contrast with the field of information technology research. Therefore, a systematic review of methodological challenges in designing and conducting RCTs within mobile health is needed.

Objective: This systematic review aimed to (1) identify and describe RCTs investigating the effect of smartphone-based treatment in adult patients with a psychiatric diagnosis, (2) discuss methodological challenges in designing and conducting individual trials, and (3) suggest recommendations for future trials.

Methods: A systematic search in English was conducted in PubMed, PsycINFO, and EMBASE up to August 12, 2019. The search terms were (1) psychiatric disorders in broad term and for specific disorders AND (2) smartphone or app AND (3) RCT. The Consolidated Standards of Reporting Trials electronic health guidelines were used as a template for data extraction. The focus was on trial design, method, and reporting. Only trials having sufficient information on diagnosis and acceptable diagnostic procedures, having a smartphone as a central part of treatment, and using an RCT design were included.

Results: A total of 27 trials comprising 3312 patients within a range of psychiatric diagnoses were included. Among them, 2 trials were concerning drug or alcohol abuse, 3 psychosis, 10 affective disorders, 9 anxiety and posttraumatic stress disorder, 1 eating disorder, and 1 attention-deficit/hyperactivity disorder. In addition, 1 trial used a cross-diagnostic design, 7 trials included patients with a clinical diagnosis that was subsequently assessed and validated by the researchers, and 11 trials had a sample size above 100. Generally, large between-trial heterogeneity and multiple approaches to patient recruitment, diagnostic procedures, trial design, comparator, outcome measures, and analyses were identified. Only 5 trials published a trial protocol. Furthermore, 1 trial provided information regarding technological updates, and only 18 trials reported on the conflicts of interest. No trial addressed the ethical aspects of using smartphones in treatment.

Conclusions: This first systematic review of the methodological challenges in designing and conducting RCTs investigating smartphone-based treatment in psychiatric patients suggests an increasing number of trials but with a lower quality compared with classic medical RCTs. Heterogeneity and methodological issues in individual trials limit the evidence. Methodological recommendations are presented.

J Med Internet Res 2019;21(10):e15362




Psychiatric disorders represent a major burden of disease worldwide with a significant impact on the quality of life, socioeconomic factors, and life expectancy [1]. In 2010, the worldwide expenses because of mental illness were estimated to be between US $2.5 trillion and US $8.5 trillion [2]. Across European countries, 27% of the adult population suffers from at least one psychiatric disorder [3]. At the same time, there is a gap between the need for treatment and the number of patients receiving treatment. The number of patients who do not receive treatment for their disorder is 35% to 50% in high-income countries and 76% to 85% and in low- and middle-income countries [4].

In 2011, the World Health Organization stated that “the use of mobile and wireless technologies to support the achievement of health objectives (mHealth) has the potential to transform the face of health service delivery across the globe” [5]. The number of smartphone users exceeded 2.5 billion people in 2018 [6], and in high-income countries, 80% of the population own and use a smartphone [7].

Smartphones are a promising tool in the field of psychiatry. They are widely used and always at hand, allowing for delivery of treatment to patients in real-time and naturalistic settings, and can augment already available treatments. At the same time, smartphones contain several sensors and technologies enabling patients, researchers, and clinicians to access information about physical and social activities [8].

Mobile health (mHealth) and especially smartphone-based technology and solutions are developing at an enormous speed, driven mainly by software and computer scientists and private companies. Thus, most available apps have not been scientifically investigated, and the validity, treatment effect, and safety have been sparingly investigated [9,10]. Nevertheless, hundreds of apps claiming to help or monitor psychiatric disorders are already available in app stores [11].

Evidence, Randomized Controlled Trials, and Interdisciplinary Research

Randomized controlled trials (RCTs) represent the methodological golden standard of excellence in medical research for the investigation of possible positive and negative effects of treatment interventions [12]. The importance of RCTs in medical science is mainly because of their ability to eliminate confounding of known and unknown nature. If properly designed and conducted, RCTs are especially useful for examination of small or moderate positive and negative effects [12].

The design and conduct of RCTs within smartphone-based treatment interventions and other electronic treatments should follow the RCT standards within medicine, while also taking into account particular challenges with electronic effect research. In RCTs, in general, there is a significant time gap from design and trial initiation to the publication of results. A recent Australian study of publicly funded clinical trials showed a median time of 7.1 years (95% CI 6.3-7.6 years) from funding to the main article on trial results being published [13]. This time gap is particularly problematic when testing smartphone-based treatment interventions as the technological development taking place within the timespan of an RCT is enormous. The technology tested is at risk of being outdated when results are published and before being taken into clinical usage [14]. Furthermore, the locked nature of treatment interventions tested in RCTs contrasts with the constantly evolving and iterative nature of app solutions [15].

Previous Reviews

Other groups have previously suggested changes to the development process, to speed up the process from idea to publication [14,16,17]. A previous review from 2013 [18] identified effect studies of smartphone-based treatments within psychiatry (including stress). This review was not limited to include RCTs, and thus, it also included pre- to posttest studies. Only 8 studies were identified despite an extensive search and broad inclusion criteria.

Symptom-specific reviews within branches of psychiatry have been done as well. As an example, Firth et al [19] found studies regarding depressive symptoms and conducted a meta-analysis on the possible effects of smartphone-based treatments on these symptoms. A total of 18 RCTs were identified; however, often, a psychiatric diagnosis was not present as the focus was on depressive symptoms. This study focused on RCTs only. The focus was on research methods and design in relation to smartphone-based treatment interventions within psychiatry. This study was limited to RCTs providing information on diagnostic measures to ensure that participants suffers from a psychiatric diagnosis, excluding trials focusing on symptoms in healthy populations.


This systematic review aimed to identify and describe all available RCTs using smartphones as (part of) a treatment intervention conducted in the field of psychiatry using sufficient diagnostic measures. Furthermore, it aimed to describe the methodology of these individual trials and discuss methodological challenges related to designing and conducting RCTs within smartphone-based treatment in psychiatric disorders using the Consolidated Standards of Reporting Trials (CONSORT) electronic health (eHealth) checklist [20] as a reference and to provide recommendations for future trials within the area.

Methods of the review and eligibility criteria were established in advance by 3 of the authors (MLT, MFJ, and LVK). Minor modifications such as adding further information to be extracted were made to the review protocol during the review process.

Trial Selection

Original trials reporting on smartphone-based treatment interventions investigated in RCTs, including adult patients with psychiatric disorders, were eligible for review. In addition, peer-reviewed articles, posters, and conference abstracts were eligible for review. If multiple articles reported on overlapping populations, the article presenting the largest population was included. No restrictions regarding sample size were applied.

The exclusion criteria were as follows: (1) children younger than 18 years; (2) psychiatric symptoms as part of somatic disorders (ie, preoperation anxiety or depressive symptoms in terminal cancer patients); (3) trials concerning stress, cigarette smoking, low intelligence quotient (IQ), and isolated sleep problems without psychiatric disorders; (4) trials with individuals who self-identified as having a psychiatric diagnosis but without diagnostic reassurance; (5) trials reporting on symptoms without diagnoses (ie, depressive symptoms or alcohol usage among college students); (6) trials using internet therapy without an active smartphone-based component (ie, if the Web page was accessible from a smartphone browser); (7) trials using only cell phones in traditional ways with text messages and phone calls (not using smartphone-based features); (8) trials using smartphones as a screen for virtual-reality setups as the primary treatment component; and (9) trials not available in English.

Information Source, Trial Selection, and Data Extraction

We conducted a systematic search covering PubMed, PsycINFO, and EMBASE on April 23, 2018, and it was last updated on August 12, 2019. Only articles from 2008 onwards were included (the time of the release of the first smartphone). References from articles and other reviews were also examined, but they did not result in any additional trials to include. The trial selection was conducted by 2 researchers (MFJ and MLT), and articles with doubt about the relevance were discussed between the 2 of them. Full-text articles for possible relevant trials were obtained if the abstract and title did not supply sufficient information. The search strategy included (1) psychiatric disorder as a broad term and for specific diseases AND (2) smartphone or app AND (3) RCT. A wide variety of text words were used to include trials published within the last 6 months that had not yet been indexed with Medical Subject Headings terms. Search strategy in PubMed was as follows:

(((Smartphone[MeSH terms] OR mobile application[MeSH Terms] OR smartphone OR mobile application OR smart phone OR mobile phone OR app OR apps OR cell phone OR Iphone* OR IOS OR Android phone OR smartphones OR mobile applications OR smart phones OR mobile phones OR cell phones)) AND (((((((((mental disorder[MeSH Terms]) OR (mental disorder OR mental disorders OR mental disease OR mental diseases OR mental diagnose OR psychiatric disease OR psychiatric diseases OR psychiatric disorders OR psychiatric disorder OR psychiatric diagnose)) OR ((drug OR substance OR prescription drug OR alcohol OR narcotic OR heroin OR amphetamine OR cocaine OR marijuana OR opioid OR morphine OR phencyclidine) AND (abuse OR dependence OR addiction))) OR (feeding disorder OR feeding disorders OR eating disorders OR eating disorder OR anorexia OR bulimia OR binge eating)) OR (autism OR autistic OR Asperger disease OR Aspergers disease) OR Asperger disorder OR Aspergers disorder OR ADHD OR attention deficit disorder OR ADD OR attention deficit hyperactivity disorder)) OR (personality disorder OR personality disorders OR obsessive-compulsive personality OR compulsive personality OR obsessive personality OR psychopath OR sociopathic OR antisocial OR passive-dependent personality OR dyssocial OR schizoid OR schizotypal)) OR (schizophrenia OR psychoses OR psychosis OR psychotic OR paranoid OR schizoaffective OR schizophreniform OR delusional)) OR (major depressive disorder OR unipolar depression OR unipolar disorder OR depressive syndrome OR endogenous depression OR neurotic depression OR melancholia OR cyclothymic OR dysthymic OR mood disorder OR mood disorders OR affective disorder OR affective disorders OR bipolar OR manic-depressive OR mania OR manic) OR (anxiety OR anxieties OR panic disorder OR agoraphobia OR obsessive disorder OR compulsive disorder OR obsessive-compulsive disorder OR phobic disorder OR phobic disorders OR PTSD OR posttraumatic stress disorder OR post-traumatic stress disorder OR post traumatic stress disorder))) AND ((randomized controlled trial[MeSH Terms]) OR (randomized controlled trial OR randomised controlled trial OR randomised OR randomized OR RCT OR randomized clinical trial OR randomiced clinical trial OR randomized clinical trial OR randomized controlled clinical trial OR randomised controlled clinical trial))

Data were extracted by using the CONSORT eHealth checklist as a template for data extraction [20]. The following data were extracted:

  • Author, year, country, trial design, trial registration, protocol availability, patients’ age and gender, sample size and use of power calculations, length of treatment intervention, and follow-up period.
  • Recruitment and diagnostics procedures of patients, recruitment length, outcome measures, well-defined hierarchy of outcome measures, and collection of outcome data.
  • Description of treatment intervention and comparator, use of blended treatment and standard treatments, affiliation with industry and technology descriptions.
  • Title according to CONSORT recommendations; use of prompts, platform choice, and possible lent-out of smartphones; economic compensation; use of placebo smartphones; methodological information regarding randomization and blinding procedures; information about the statistical approach to technical updates and whether updates and technical crashes or down periods were reported; possible harms; adherence to the smartphone system; and baseline data on patients’ technological skills and funding information.

A data extraction template is provided in the supplementary material (Multimedia Appendix 1). The extracted data are presented in 4 tables. Data were initially retrieved by one author (MLT) and subsequently and independently by another author (MFJ). Any disagreements were solved between MLT and MFJ.

Three tables describe the trials individually, with various focus. The fourth uses the CONSORT eHealth checklist [20] as a template to summarize the relevant findings according to these in a systematic way and hereby partly summarizes the relevant findings from previous tables, but it also includes other new relevant information. The tables are presented in relevant sections in the result-section.

Trial Selection

Figure 1 presents a flow diagram represented according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines [21], showing the results of the literature search and selection of trials. The initial literature search in PubMed, EMBASE, and PsycINFO on April 23, 2018, resulted in a total of 1490 articles. Of these, 380 duplicates were removed, resulting in 1110 articles. Furthermore, 833 articles were excluded based on the title and year of publication with the main reasons for exclusion being as follows: not concerning psychiatric disorders (eg, HIV, Alzheimer disease, and obesity), not reporting on results from RCTs, and published before 2008. This led to a total of 277 remaining articles, from which abstracts were examined.

Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram displaying information on article flow from initial search to final inclusion. IQ: intelligence quotient; RCT: randomized controlled trial.
View this figure

Out of these 277 articles, 239 were excluded based on abstracts, with majority of reasons being as follows: trial protocols, text message or other nonsmartphone technology, publications on same data (the publication with the most data was included), measuring on symptoms—not diagnosis, no or insufficient diagnostic reassurance, participants younger than 18 years, topics concerning low IQ, smoking, stress, or isolated sleep problems.

This led to 38 articles that were printed for full-text reading. Among them, 19 articles were subsequently excluded because of the following: reporting symptoms, not disorders (n=7); no or insufficient diagnostic reassurance (n=6); only abstract in English (n=1); insomnia (n=1); no outcome effect measures (n=1); and not using smartphone as part of treatment (ie, only using smartphones to reach the patient or allowing patients to answer emails and questionnaires on smartphone; n=3). References from articles and other reviews were also examined, but they did not result in any additional trials to include, giving a total of 19 eligible articles from the initial search.

The search was updated on August 12, 2019, resulting in 8 new articles. Thus, a total of 27 unique trials [22-48] were identified and included in this review, including 5 with diagnostic estimates based on questionnaires using relevant cutoff scores [44-48]. Included trials are described individually in the tables.

Trial Characteristics

Across all included trials, a total of 3,312 patients were included. The included trials represented a range of psychiatric disorders. Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (comprising 3 on bipolar disorder and 7 on unipolar depressive disorder), 9 on anxiety and posttraumatic stress disorder posttraumatic stress disorder (F40-F48) [23,30,34,39,43,44,46-48], 1 on eating disorders (F50-F59) [31], and 1 on attention-deficit disorder (F90-F98) [29]. One trial included patients with severe mental disorders across ICD-10 diagnoses [33].

Most trials had an overrepresentation of females, except trials concerning veterans, schizophrenia, or substance or alcohol abuse. Patients’ ages ranged from 18 to 73 years. In the majority of the trials, the average age was 40 years.

Trial Design and Reporting

The number of RCTs testing smartphone solutions in patients with a psychiatric disorder increased with time, especially from 2018 onward. Some trials [24,27,29-33,35,40] were leaning toward traditional RCT designs conducted in clinical settings and using the CONSORT checklist for reporting and designing trials [20]. Some other trials did not provide clear information regarding these issues, leaving the reader with lacking information on the design and conduct of the trial.

Overall, 11 trials had a sample size above 100 patients [24,30,32-34,39-41,44,45,48]. One trial [40] had been repeated using the same intervention, trial design, and outcome measures as a previous trial [27].

A total of 14 articles included information about trial registration, such as the [22,24-27,29-33,35,39-41]. For the remaining trials (including the study by Krzystanek et al [41], where we could not find any information on the registration information provided) it was not possible to see whether the primary outcome measure was predefined in the primary hypothesis leading to the trial design. In addition, 5 trials referred to a published trial protocol in the article [27,30,32,35,40]. One additional article attached the study protocol in the supplementary section when published [28].

Information about the individual trial designs, registration, sample characteristics, and length of intervention and follow-up is presented in Table 1.

Table 1. Randomized controlled trials involving smartphones in the field of psychiatry identified in systematic search updated in August 2019. Description of basic information and trial design. The bottom 5 trials in italics are trials with diagnoses solely based on questionnaires.
Author, year of publicationCountryTrial designProtocolaFemale/totalAnalyzed (power calculation)Age (years), mean (SD)Intervention length (weeks)Posttreatment follow-up (weeks)
Watts et al, 2013 [22]AustraliaPilot RCTbN/Ac28/3525 (N/A)41 (12.4)812
Dagöö et al, 2014 [23]SwedenRCTN/A39/5752 (N/A)52912
Gustafson et al, 2014 [24]United StatesRCTN/Ac137/349349 (350)38 (10)3216
Ly et al, 2014 [25]SwedenRCTN/Ac57/8181 (N/A)36.0 (10.8)816
Depp et al, 2015 [26]United StatesRCTN/Ac48/8282 (N/A)47.51014
Faurholt-Jepsen et al, 2015 [27]DenmarkRCTPublishedc45/6778 (56)18-60d24No
Ly et al, 2015 [28]SwedenNIe RCTAttached65/9593 (93)18-73d924
Moëll et al, 2015 [29]SwedenRCTN/Ac39/5757 (N/A)36.8 (10.9)6No
Ivanova et al, 2016 [30]SwedenRCTfPublishedc98/152152 (150)35.31052
Hildebrandt et al, 2017 [31]United StatesRCTN/Ac55/6666 (80)32.11224
Mantani et al, 2017 [32]JapanRCTPublishedc87/164164 (164)25-59d817
Ben-Zeev et al, 2018 [33]United StatesRCTN/Ac67/163163 (160)49 (10)1212
Boettcher et al, 2018 [34]Sweden/GermanyRCTfN/A161/209209 (N/A)35.4 (12.4)1252
Bucci et al, 2018 [35]EnglandPilot RCTPublishedc18/3636 (N/A)N/A1210
Hur et al, 2018 [36]Republic of KoreaPilot RCTN/A26/4834 (N/A)≈243No
Liang et al, 2018 [37]ChinaPilot RCTN/A21/7475 (N/A)41.6 (8.0)4No
Schlosser et al, 2018 [38]United StatesRCTN/A15/4343 (N/A)241212
Stolz et al, 2018 [39]SwitzerlandRCTfN/Ac94/150150 (141)351212
Faurholt-Jepsen et al, 2019 [40]DenmarkRCTPublishedc76/129129 (117)43 (12.4)36No
Krzystanek et al, 2019 [41]PolandRCTN/Ac116/290Varying32.1 (6.2)52No
Stiles-Shields et al, 2019 [42]United StatesPilot RCTN/AN/A/3027 (N/A)N/A64
Teng et al, 2019 [43]TaiwanRCTfN/A61/8282 (N/A)≈21.544
Enock et al, 2014 [44]United StatesRCTN/A205/429326g (N/A)18-68d48
Roepke et al, 2015 [45]United StatesRCTfN/A197/283283 (207)40.2 (12)42
Miner et al, 2016 [46]United StatesPilot RCTN/A40/4949 (N/A)45.7 (13.9)44
Possemato et al, 2015 [47]United StatesPilot RCTN/A1/2020 (N/A)428No
Kuhn et al, 2017 [48]United StatesRCTN/A83/120120 (120)391212

aMentioned in the article.

bRCT: randomized controlled trial.

cTrial is registered.

dAge interval (mean not given in the article).

eNI: noninferiority.

fThe trial had 3 arms.

gIncluded in the analysis if score is greater than cutoff.

Settings and Diagnostic Procedures

A total of 8 trials were conducted in traditional clinical settings, with patients referred by clinical staff or contacted in their treatment clinic [24,27,32,33,35,37,40,41]. Others used remote designs where patients who self-identified as having a psychiatric disorder applied for participation [22,23,25,26,28,30,34,38,39,42-44,46,48]. None of the trials compared participants with nonparticipants. All but 2 trials [43,44] presented a flowchart of eligible subjects with varying details on reasons not to participate. Trials with open recruitment (eg, internet forums, Web pages, Facebook, and advertising) [22,23,25,26,28-30,34,36,38,39,42-46,48] only had information on people who signed up. Completion rates were reported very differently and varied from 163 patients out of 164 (99.4% [163/164]) completing the primary outcome [32] to 74 out of 283 patients (26.1% [74/283]) [45]. In addition, 12 trials compensated for participation in trial assessments with money or gift cards [26,33-38,42,43,46-48].

Validation and certainty of diagnosis varied substantially, and often, a pragmatic setup was used, leaving the validity of the obtained diagnosis with uncertainty.

A total of 5 trials based their diagnosis solely on clinical-based information [24,33,35,37,41]. Furthermore, 15 trials used research-based diagnoses without clinical information, using either questionnaires [44-46,48], phone calls [22,23,25,28,34,42], video interviews [38], or personal interviews [31,36,39,43] to validate the self-reported diagnoses. In addition, 7 trials included patients with a clinical diagnosis that was subsequently assessed and validated by the researchers using questionnaires [47], phone calls [29,30], or personal interviews [26,27,32,40].

Hypotheses and Use of Predefined Hierarchical Outcome Measures

Overall, 18 trials included clearly described hypotheses [22,24-28,30-32,34,36,38-40,44,45,47,48], and 8 trials [22,37,41-44,46,47] did not distinguish between primary and secondary outcome measures in the article and had no hierarchy of outcomes.

Outcome Measures

A total of 8 trials included objective or observer-based primary outcome measures: One trial tested the levels of drug use in the urine as a specific primary outcome measure [37]. Another trial used objective measurements of feasibility, use, and attrition as the primary outcome [35]. One trial tested a specific task of motivated behavior [38]. In addition, 4 trials used scores on clinical ratings as the primary outcome measure assessing the level of depression and mania [26,27,40], levels of bulimic episodes [31], or psychotic symptoms [41]. The remaining trials used patient-reported outcome measures.

Within trials of similar diagnoses, authors used different scales and measures for the primary outcome measure. For instance, in measuring depression scores, the following outcome measures were used as the primary outcome: Patient Health Questionnaire, Beck Depression Inventory, Hamilton Rating Scale for Depression 17 item, Montgomery–Åsberg Depression Rating Scale, and Center for Epidemiologic Studies Depression Scale [22,25-28,32,33,36,40,42,45].

Furthermore, 2 trials, with apps for training in attention bias modification, claimed to be double blinded, with no further explanation on how blinding was ensured [43,44]. One trial blinded app allocation for patients when testing 2 different apps [45]. The remaining trials did not blind patients for the intervention (active vs control). In 12 trials, the authors explicitly stated that they had used blinded assessments for outcome measures [25-29,31-33,35,38,40,44]. Within these 12 trials, 5 trials used the patient-reported outcome as the primary outcome measure in nonblinded patients [25,28,29,32,33]. Although blinded assessors collected these data, answers to the questionnaires were self-reported. One trial tested for the success of blinding [32].

Information about recruitment, diagnosis, outcome measures, and how these were obtained can be found in Table 2.

Table 2. Randomized controlled trials involving smartphones in the field of psychiatry identified in systematic search updated in August 2019. Description of participant characteristics and outcome data. The bottom 5 trials in italics indicate trials with diagnoses solely based on questionnaires.
Author, year of publicationDiagnosisHow was the diagnosis obtained?Recruitment: open/closed (recruitment length in months); informationPrimary outcomeaQuestionnaire data collection
Watts et al, 2013 [22]Major depressive disorder (Rb)MINIc phone interview + PHQ-9dOpen (3 months); advertising + application on a Web pageQuestionnaire(PHQ-9, BDI-IIe, K-10f, and other)N/Ag
Dagöö et al, 2014 [23]Social anxiety disorder (R)MINI phone interviews SCIDhOpen (2011 and 2012); advertising in media and Facebook.Questionnaire (LSAS-SRi)aInternet platform
Gustafson et al, 2014 [24]Alcohol use disorder (Cj)From treatment centers (DSM-IVk)Closed (17 months); from 3 nonprofit residential treatment centersRisky drinking daysaPhone interview
Ly et al, 2014 [25]Major depressive disorder (R)MINI phone interviewOpen (N/A); advertising in national mediaQuestionnaires (PHQ-9 and BDI-II)aInternet platform
Depp et al, 2015 [26]Bipolar disorder (R+C)Medical records + MINI interviewOpen (N/A); online, self-help groups, outpatient clinics, and communitiesClinical ratings (MADRSl and YMRSm)aN/A
Faurholt-Jepsen et al, 2015 [27]Bipolar disorder (R+C)From outpatient clinic + SCANnClosed (18 months); recruitment from specialized hospital functionClinical rating (Hamilton Rating Scale for Depression 17 item and YMRS)aPaper
Ly et al, 2015 [28]Major depressive disorder (R)MINI phone interview + PHQ-9Open (N/A); advertising in media (for patients’ self-identifying as depressed)Questionnaires (BDI-II)aInternet platform
Moëll et al, 2015 [29]Attention-deficit/hyperactivity disorder. (R+C)Medical records + DSM-IV phoneOpen (N/A); by patient websites and FacebookQuestionnaire Adult Self-Reported Scale subscaleaInternet platform
Ivanova et al, 2016 [30]Social anxiety/panic disorder (R+C)MINI phone + questionnairesOpen (2 months); advertising in national mediasQuestionnaire (Generalized Anxiety Disorder 7-item, LSAS-SR, + more a)aInternet platform
Hildebrandt et al, 2017 [31]Binge eating and bulimia (R)SCID interview + questionnairesBoth (N/A); community advertising and referralsClinical ratings (objective bulimic episodes Eating Disorder Examination)aPaper and in-app
Mantani et al, 2017 [32]Major depressive disorder (R+C)Personal by treating physicianClosed (25 months); recruited by treating physiciansTelephone (blinded; PHQ-9)aTelephone assessment
Ben-Zeev et al, 2018 [33]Severe mental illnesso (C)Chart diagnosisClosed (27 months); identified from medical records, recruited by clinical teamsQuestionnaire (Symptom Checklist-9) Engagement (objective)aN/A
Boettcher et al, 2018 [34]Social anxiety disorder (R)DSM-IV (phone)Open (N/A); advertising in national media, FacebookQuestionnaire (LSAS-SR)aInternet platform
Bucci et al, 2018 [35]Early psychosis (C)From outpatient clinicClosed (7 months); From early intervention for psychosis serviceObjective measurements of feasibility and attritionaN/A
Hur et al, 2018 [36]Depression (other)p (R)SCID non-patient interview + questionnairesOpen (N/A); advertising, online recruitment, posters and clinicQuestionnaires (Dysfunctional Attitude Scale)aN/A
Liang et al, 2018 [37]Substance use disorder (C)From methadone treatment clinicsClosed (11 months); from methadone maintenance clinics and via social workersRobust objective measure (drug use measured in urine)Interviews
Schlosser et al, 2018 [38]Schizophrenia spectrum disorders (R)DSM-IV video-interviewOpen (N/A); Craigslist, advertising, and information boardsMotivated behavior measured by trust task (objective task)aInternet platform
Stolz et al, 2018 [39]Social anxiety disorder (R)DSM-IV (master students)Open (N/A); media and online forumsQuestionnaires (Social Phobia Scale, SIASq, LSAS-SR)aInternet platform
Faurholt-Jepsen et al, 2019 [40]Bipolar disorder (R+C)SCANClosed (N/A); patients previously treated in specialized function invitedClinical ratings (HDRS-17 + YMRS)aPaper
Krzystanek et al, 2019 [41]Paranoid schizophrenia (C)N/A (enrolled from treatment centers)Closed (7 month); enrolled from 27 treatment centersMany outcomes; clinical ratings (video; eg, Positive and Negative Syndrome Scale)N/A
Stiles-Shields et al, 2019 [42]Depression (R)Quick Inventory of Depressive Symptomatology + MINI phone interviewOpen (5 month); CraigslistQuestionnaires (eg, PHQ and usability)Internet platform
Teng et al, 2019 [43]Generalized Anxiety Disorder (R)Questionnaire + DSM-IV subscaleN/AQuestionnaires (State-Trait Anxiety Inventory, BDI-II + more)N/A
Enock et al, 2014 [44] Social anxiety disorder (R)Questionnaire with cutoffOpen (40 months); news articles, message boards,, GoogleQuestionnaire (LSAS-SR, SIAS-17, and other)Internet platform
Roepke et al, 2015 [45] Depression (R)Questionnaire CES-Dr above 16Open (5 months); Website and CraigslistQuestionnaire (CES-D at posttest)aInternet platform
Miner et al, 2016 [46] PTSDs (R)Questionnaire PCL-Ct >30Open (16 months); flyers, websites, CraigslistQuestionnaire (PCL-C) + feasibility and acceptabilityInternet platform
Possemato et al, 2016 [47]PTSD (R+C)Screened for PTSD + PCL >40Closed (4 months); from veteran care unit, screened for PTSD.Feasibility metrics + Questionnaires (PCL-Su, PHQ-9)N/A
Kuhn et al, 2017 [48]PTSD (R)Questionnaires PCL-C>34Open (10 months); flyers, media, social media, CraigslistQuestionnaire (PCL–C)aInternet platform

aWell-defined hierarchy in outcome measures.

bR: research based.

cMINI: Mini International Neuropsychiatric Interview.

dPHQ-9: Patient Health Questionnaire-9.

eBDI-II: Beck Depression Inventory.

fK-10: Kessler Psychological Distress Scale.

gN/A: not applicable.

hSCID: Structured Clinical Interview.

iLSAS-SR: Liebowitz Social Anxiety Scale—self-reported.

jC: clinical based.

kDSM: Diagnostic and Statistical Manual of Mental Disorders.

lMADRS: Montgomery–Åsberg Depression Rating Scale.

mYMRS: Young Mania Rating Scale.

nSCAN: Schedules for Clinical Assessment in Neuropsychiatry.

oSchizophrenia, schizoaffective disorder, bipolar disorder, and major depressive disorder.

pOther specified depressive disorder.

qSIAS: Social Interaction Anxiety Scale.

rCES-D: Center for Epidemiologic Studies Depression Scale.

sPTSD: posttraumatic stress disorder.

tPCL-C: Post-Traumatic Checklist—Civilian.

uPCL-S: Post-Traumatic Checklist Scale.


Although the included trials shared the concept of smartphones as a core feature of their interventions, the interventions were very different and tested in diverse settings with heterogeneous patient groups. Generally, 8 trials tested the app as a stand-alone treatment [35-37,44-48]. The remaining trials used variations of blended treatment and clinical support. Overall, 18 trials used prompts to engage users either from the app or by the investigators [22,26-28,30,32-44].

A glimpse of the diversity of interventions is presented here. Some trials used smartphones as a part of internet-based therapy, whereas others used smartphones as augmentation for face-to-face therapy or standard treatment programs. Most trials developed specific apps, whereas others used either commercially available or previously developed apps. Some interventions were interactive, whereas others had more static content. Some of the interventions made use of the unique possibilities a smartphone represents such as using global positioning system or allowing to interact with peers, relatives, or professionals, whereas others resemble classic internet therapies made for small smartphone screens to be more convenient and accessible to patients.

Control Group

One of the included trials involved the use of a placebo app, consisting of an inactive version of the software with limitations, but with no further description [41]. In addition, 2 trials (that both tested the effect of attention bias modification via a dot probe on the smartphone screen) used a different version of the app for the control group, setting the dot appearance at random instead of following a predefined pattern that was central to the treatment [43,44]. In 1 trial on bipolar disorder, the participants in the control group received a placebo smartphone without the app system [27]. No other trials mentioned attempts on placebo treatment. Overall, 5 trials used standard treatment as the comparator [24,27,35,38,40], and 11 trials used a waitlist control group [29,30,34,38,39,41,42,44-46,48]. Furthermore, 4 trials used some sorts of clinical intervention as the comparator [26,28,31,33], and 5 trials compared the intervention with another app [23,25,35,36,42].

Further description of the smartphone-based treatment interventions, availability of technology, and author affiliation with industry can be found in Table 3 and is summarized in Table 4.

Table 3. List of randomized controlled trials involving smartphones in the field of psychiatry identified in systematic search updated in August 2019. Description of intervention and control group as well as authors cooperation with the industry. The bottom 5 trials in italics indicate trials with diagnoses solely based on questionnaires.
Author, year of publicationShort description of the intervention and main components. If available, the app name is displayed in italics.Comparator. treatment received by the control groupsBlended treatment (BT)/app alone (AA)TAUaCooperation/affiliation with the industrybDescription of technology available for the reader
Watts et al, 2013 [22]CBTc-based “get happy program” with comic book–like lessons + homework activitiesPC version of the same programBT (limited clinician contact)N/AdNoBrief description and few screenshots
Dagöö et al, 2014 [23]Guided internet-based CBT adapted for mobile phone administrationAnother app similar therapist contactBT (limited clinician contact)NoN/ABrief descriptions
Gustafson et al, 2014 [24]A-chess: app with static self-help content and interactive features with therapist feedbackTAUBTYesNoApp fully available online and a description of the app is attached
Ly et al, 2014 [25]App delivering behavioral activation psychotherapy with possible but limited clinician contactMindfulness app, similar therapist contactBT (limited clinician contact)NoThe first author has a similar app on the open marketGood descriptions and screenshots
Depp et al, 2015 [26]PRISM: interactive monitoring and intervention linking mood and activities with self-management strategiesActive control monitoring on paperBTN/ANoThorough descriptions but no technical reports or screenshots
Faurholt-Jepsen et al, 2015 [27]MONARCA: self-monitoring with a double feedback loop between clinic (nurse) and patientTAU + nurse contact + phone without appBTYesNoThorough descriptions and screenshots in the protocol
Ly et al, 2015 [28]Four therapy sessions and a smartphone app, based on behavioral activation, used between sessionsFull behavioral activation (10 sessions)BTNoN/ABrief descriptions and screenshots
Moëll et al, 2015 [29]Living smart: Guided online course to structure life using multiple already available appsWaitlist controlBTNoN/AMultiple already-available apps
Ivanova et al, 2016 [30]Internet therapy + an appe promoting change corresponding to the core treatment program, with therapist supportWaitlist or intervention without therapist supportBTNo2 authors employed by a technology company; 1 developed a similar appDescription and screenshots available in the protocol
Hildebrandt et al, 2017 [31]NOON self-monitoring: App as an augmentation of traditional guided self-helpGuided self-help therapy without an appBTNo3/5 authors have a connection to NOOM who developed the appShort descriptions and no screenshots
Mantani et al, 2017 [32]Kokoro: CBT-based self-help app with 8 sessions presented by cartoons + fixed-dose medicine shiftMedicine shift with fixed dose and no appBTfNo2 of the authors developed the appA thorough report describing the app in detail
Ben-Zeev et al, 2018 [33]FOCUS: Multimodal smartphone intervention including self-assessments and on-demand functionsClinic-based group interventionBTNoFirst author had a consulting agreement with technology companyA short description in the text - further in supplement
Boettcher et al, 2018 [34]Challenger: App promoting exposure exercise through interactive challenges + internet-based therapy2) Waitlist control or 3) internet therapy aloneBTNoThe third author founded the app companyA short description + referral to further information
Bucci et al, 2018 [35]Actissist: Self-help app that asks questions and has automated responses and various static supportive contentTAU + another appAAYesN/ADescriptions of app and screenshots
Hur et al, 2018 [36]TODAC: A scenario-based CBT app to reduce dysfunctional beliefsMood diary appAAN/ANoDescriptions of the app modules, and small screenshots
Liang et al, 2018 [37]S-health: Simple smartphone app that sends messages, controls cravings, and has a surveyReceiving text messages about various topicsAAYesN/ASurvey and screenshots available + short description
Schlosser et al, 2018 [38]Prime: Personalized real-time intervention for motivational enhancement. App-based online communityTAU/waitlist controlBTYesNoShort description, but no screenshots
Stolz et al, 2018 [39]Mobile version of validated psychoeducative self-help program with 8 modules based on cognitive therapyWaitlist control or PC versionBT (limited clinician contact)NoN/AShort description of modules, but no screenshots
Faurholt-Jepsen et al, 2019 [40]MONSENSO: Self-monitoring + objective monitoring with a double feedback loop between nurse and patientTAU + offer to borrow a smartphoneBTYes2 coauthors are shareholders in MonsensoThorough descriptions and screenshots in the protocol
Krzystanek et al, 2019 [41]MONEO: Medication reminder, cognitive training, information bank, and “tele visits” with the investigatorInactive version + monthly video examinationBTN/ANoRefers to online supplementary that was not possible to find
Stiles-Shields et al, 2019 [42]Boost Me (behavioral app) or Thought Challenger (a cognitive app): Both with brief weekly coaching2 different apps and 1 waitlist controlBT (with limited coaching)NoLast author has an ownership interest in Actualize TherapyBoth apps are available free online
Teng et al, 2019 [43]Home-delivered attention bias modification training with dot probe on screenControl group with random dot or waitlistBTN/ANoShort descriptions and few screenshots
Enock et al, 2014 [44] Cognitive training via smartphone with attention bias modification trainingAn active control group and waitlist controlAAYesNoThorough descriptions, links, and few screenshots
Roepke et al, 2015 [45] SuperBetter: Self-help game using either specific CBT or a general version using self-esteem and acceptance2 versions of the app and 1 waitlist controlAAYes3 authors work for SuperBetter (1 founded it)Short description and 2 screenshots
Miner et al, 2016 [46] PTSD coach: Psychoeducation, symptom assessments, self- management + access to supportive othersWaitlist controlAAN/AN/AThe app is available free online
Possemato et al, 2016 [47] PTSD coach: Multifunctional psychoeducative self-help app with clinical support of 4, 20-min sessionsApp aloneBT<=>AANoNoThe app is available free online
Kuhn et al, 2017 [48] PTSD coach: psychoeducation, symptom assessments, self-management + access to supportive othersWaitlist controlAANoN/AThe app is available free online

aTAU: Treatment as usual.

bInformation assessed by author affiliation, grand support, and conflict of interest.

cCBT: cognitive behavioral therapy.

dN/A: not applicable.

eTwo treatments not technologically attached but based on same therapy.

fAllowed to discuss the app with treating physician.

Table 4. Summarized findings from systematic review on 27 randomized controlled trials involving smartphones in the field of psychiatry. Consolidated Standards of Reporting Trials electronic health checklist is used as a guideline to systematically display trial design, methodology, and reporting of the identified trials.
Consort itemSummarized findings according to CONSORT item, with references to relevant articles.
Title and abstract (1a and 1b)All but 6 titles [22,35,37,38,41,42] described the mode of delivery, components of treatment, target group, and trial design according to Consolidated Standards of Reporting Trials electronic health guidelines [20]. Often only broad terms of components were used, such as “mobile” or “mHealth.”
Introduction (2a and 2b)The trials were published from 2013 to 2019 with equal distribution through 2013 to 2017 and increasing numbers from the from 2018 and forward [33-43]. Trials were mainly from western countries, especially from Scandinavia [23,25,27-30,34,40].
Trial design (3)A total of 19 trials were classic RCTs [23,25-27,29-34,38-41,43-45,48], 7 were pilot RCTs [22,35-37,42,46,47], and 1 was a noninferiority RCT [28].
Participants (4a and 4b)A total of 22 trials used research-based diagnoses: 5 were based on questionnaires [44-48], 8 used phone interviews [2225,28-30,34,42], 1 used Facetime or Skype interview [38], and 8 used personal interviews [26,27,31,32,36,39,43] mostly based on either MINI or DSM-4; 5 trials based their diagnoses only on clinical-based information [24,33,35,37,41]; 13 trials excluded patients with various degree of suicidal ideation [22,23,25,28-32,34,36,39,42,47]; 3 trials excluded patients with too severe symptomatology within the diagnosis of interest [22,26,27]. Most trials excluded patients with severe psychiatric comorbidity from lower International Statistical Classification of Diseases and Related Health Problems-10 chapter. In addition, 12 trials supplied participants with smartphones, either voluntary or mandatory [24,26,27,31,33,35,38,40,41,46-48]; 12 trials compensated participation in assessments with money or gift cards [26,33-38,42,43,46-48].
Interventions (5)Intervention length varied substantially: from 3 weeks [36] to 52 weeks [41]. Most interventions lasted between 4 and 12 weeks; 8 trials used unaffected standard treatment beside intervention [24,27,35,37,38,40,44,45]; 8 trials tested the app alone [35-37,44-48], the remaining used variations of blended treatment; 1 trial tested blended therapy against app alone [47]; 18 trials used prompts to engage users, either from the app or by investigators [22,26-28,30,32-44]; 1 trial compared with an inactive “placebo” version of the app [41]; 2 trials compared with a placebo training module [43,44]. In 1 trial, participants in the control group received a “placebo” smartphone without the app system [27]; 5 trials used standard treatment as comparator [24,27,35,38,40], 11 trials used waitlist control [29,30,34,38,39,42-46,48], 4 trials used clinical intervention [26,28,31,33], and 5 trials compared with another app [23,25,35,36,42]; 1 trial collected automatically generated data [40] and further 8 trials collected data on app usage [24,33,34,42,43,45,47,48]; in 3 trials, intervention was only for iPhone [32,34,45] and in 3 only for Android [27,42,43]. The rest of the trials either had a Web-based version available or app for both platforms. Only 1 article mentioned information about updates of apps or intervention [27].
Outcomes (6a and 6b)Overall, 8 trials did not use a predefined hierarchy of outcome measures [22,37,41-44,46,47]; 1 trial used tested levels of drug use in urine as a specific detection [37]; 1 trial used objective measurements of feasibility, use, and attrition as the primary outcome [35]; 1 trial tested a specific task [38]; 1 trial used video call–based clinical ratings [41]. Only 4 trials used clinical ratings as the primary outcome [26,27,31,40]. Remaining trials used patient-reported outcome measures. A total of 12 trials used internet platform for data collection [23,25,28-30,34,38,39,42-46,48], with 6 of these mentioning validations of questionnaires for online use [23,25,28,30,34,46]; 2 trials used the app for outcome measure [31,41].
Sample size (7a and 7b)Sample size varied from 20 participants [47] to 429 participants [44]; 11 trials with numbers above 100 participants [24,30,32-34,39-41,44,45,48]. Pilot trials were smaller.
Randomization (8, 9, and 10)Overall, 8 trials did not supply information about randomization [29,31,36-38,45-47]; 1 used Excel [43], and 1 used the app for randomization [41]. The remaining mainly used online software.
Blinding (11a and 11b)Overall, 2 trials claim to be double-blinded with no further explanation on how blinding was assured [43,44]; 1 trial blinded app allocation for the patients (they tested 2 different apps) [45]. The remaining trials had no blinding of patients. In 12 trials, authors explicitly wrote that they used blinded assessments for outcome measures [25-29,31-33,35,38,40,44]. Within these 12 trials, 5 trials used patient-reported outcome measures as the primary outcome measure with nonblinded patients [25,28,29,32,33]; 1 trial tested for the success of blinding [32].
Statistical methods (12a and 12b)A total of 11 trials based sample size on power calculations [24,27,28,30-33,39,40,45,48]. All but 1 of these managed to recruit at least the desired number [31]. No trials took changes and updates of software or technical problems into account in statistical methods.
Participant flow (13a and 13b)All but 2 trials [43,44] presented a trial flow chart of eligible subjects, although with various details on reasons not to participate and drop out. Completion rates were reported very differently, varying from 163/164 completing primary outcome [32] to 74/283 completing posttreatment assessments (primary outcome) [45]. All but 2 trials reported on adherence to treatment [36,43].
Recruitment (14a and 14b)Recruitment length was reported in 16 trials and varied from a few months to several years [22-24,27,30,32,33,35,37,41,42,44-48]; 10 trials used closed recruitment with a referral from clinicians or researchers seeking out participants from a well-defined patient population [24,27,31-33,35,37,40,41,47]; 1 trial gave no information on recruitment [43]. The remaining trials used open recruitment mainly via, advertising in traditional ways, or through social media.
Baseline data (15)Only 2 trials included technology-specific baseline data or information about participant technological abilities [32,33].
Numbers analyzed (16)All but 6 trials [22,36,37,41-43] used the intent-to-treat principles in the primary analysis.
Outcome and estimation (17)A total of 17 trials presented intensity of use or user data, either in the article or in supplementary data, with significant variations in usage among subjects and between trials [22-25,27,32-35,37-40,42,43,45,47].
Harms (19)Overall, 5 trials prospectively measured harms or adverse events and reported directly in paper [32,33,35,41,42]. These trials found no harm from smartphone treatment used. One of these trials [42] had a safety protocol with clear, standardized instructions on how to react to suicidal ideation; 1 trial found a negative effect of treatment in secondary analysis, indicating fewer improvements in symptoms in a subgroup with a higher baseline score on the Hamilton Rating Scale compared with controls [27].No trials mentioned privacy breaches. Three trials mentioned technical problems and how these affected the intervention [23,40,48].
Generalizability (21)Trials were heterogeneous. Some had strict criteria on diagnosis, comorbidity, and ongoing treatment, whereas others were pragmatic trials with few exclusion criteria. Trial populations varied from patients recruited among the general population who might not have sought help in the regular treatment system [22,23,25,28,30,34,39,42,44], whereas others came from specialized clinical functions setups [24,27,31-33,35,37,40,41,47].
Registration (23)A total of 14 articles included information about trial registration [22,24-27,29-33,35,39-41].
Protocol (24)A total of 5 trials published their trial protocol [27,30,32,35,40]; 1 trial had the protocol attached to the publication [28].
Funding (25)Most authors came from universities; 15 trials reported information regarding funding [24-28,31,32,35-38,40,41,43,47] with funding mainly coming from public funds and institutions.
Competing interest (X27)aA total of 9 articles declared having various degree of affiliation with private technology companies or closed relation to the app that they tested [25,30-34,40,42,45]; 8 trials did not include conflicts of interest in the printed article [23,28,29,35,37,39,46,48].

aNot an original Consolidated Standards of Reporting Trials item but included in the Consort electronic health checklist as X27.

Adherence to Smartphone Intervention

All but 2 trials [36,41] reported adherence by using one of the following or a combination: report on how many in-app lessons, modules or sessions participants completed, the number of daily or weekly users or logins, numbers of active users at a given time point, and composite scores measuring adherence.

Some trials included large tables on the usage of different components of the app, for example, the study by Schlosser et al [38], or included detailed program usage in the supplementary section, for example, the study by Stolz et al [39].

One trial economically compensated the participants for using the app [35]. Further comparison of adherence between trials was not feasible because of significant differences in how adherence was collected and reported in the individual trials.

Statistical Power Analysis

Overall, 11 trials based their required sample size on power analyses [24,27,28,30-33,39,40,45,48]. All but 1 [31] of these trials managed to recruit at least the desired number. In addition, 6 of the trials without power analyses were specified as pilot studies [22,35-37,42,46,47].

Statistical Analysis

All but 6 trials [22,36,37,41-43] used intent-to-treat principles in the primary analyses. Different methods were used to account for missing data. Some trials used multiple imputations [23,46,48], whereas others used mixed models [22,24,27,28,30,32,34,39,40,45] with missing at random or maximum likelihood estimations. One trial omitted data without imputation [41]. In addition, 8 trials included no explicit information on how the authors handled missing data [33,35-38,42,44,47]. Furthermore, 4 trials referred to an available predefined plan of analyses in the statistical section [27,32,33,35].

Technical Aspects of Smartphone-Based Treatment Interventions

One article informed about technical updates of the app, changes in interventions because of improvements, or technological problems that might have influenced the intervention over time [27]. None of the trials mentioned privacy breaches. In addition, 3 trials mentioned technical problems and how these affected interventions and the main hypothesis [23,40,48]. One trial collected automatically generated smartphone data (phone usage, social activity, and mobility) [40]. Furthermore, 8 trials collected data on app usage [24,33,34,42,43,45,47,48]. None of the trials accounted for changes or updates in the software or technical problems related to the intervention in the statistical analyses. Two trials included technology-specific baseline data or information about the participant’s technological abilities such as prior smartphone use or assessments of the participant’s ability to use a smartphone [32,33]. A total of 17 trials presented the intensity of use or user data with significant variations in usage among subjects and between trials [22-25,27,32-35,37-40,42,43,45,47].

Ethical Aspects

None of the included trials addressed the potential ethical aspects of using smartphones in the treatment of patients with a psychiatric disorder. Two trials included a section on ethics, including information on various trial registrations and approvals, data storage, and economic compensation [27,40].

Principal Findings

This is the first systematic review regarding methodological challenges in RCTs investigating smartphone-based treatment interventions in patients with a psychiatric diagnosis. We included 27 trials with a wide range of psychiatric diagnoses and observed substantial between-trial heterogeneity. The trials were conducted in diverse settings and used different smartphone-based treatment interventions and different follow-up periods. The trials reported on various outcome measures, which, in nearly half of the trials, were not clearly predefined. Most trials only used unblinded patient-evaluated outcome measures. A single trial reported on the success of blinding procedures.

Furthermore, only 1 trial provided information regarding technological updates of the smartphone-based treatment intervention. A declaration of interests was missing in 9 of the trials. No trial compared participants with nonparticipants, thereby increasing the risk of selection bias and making generalization of the trial findings difficult.

The included trials used smartphones in various ways and applied treatments to very heterogenic populations. Generally, the combination of insecure diagnoses, lack of blinding, use of patient-evaluated outcome measures, and lack of trial protocols or thorough publicly available trial registrations implies that evidence on the effects and side effects is still warranted.

The included trials used very different comparators, and the treatments given, besides the intervention of interest, varied from nothing to intensive clinical standard treatment setups. Generally, important aspects concerning technological features and how these inevitably affect outcomes were sparingly reported. On the basis of the results of the review, in the following sections, we discuss the highlights and suggest recommendations for designing and conducting future RCTs investigating the effect of smartphones in psychiatry.

Inclusion Criteria of Trial Patients

Owing to the lack of diagnostic biomarkers within psychiatry, currently, the research-based clinical diagnostic process represents the golden standard. The use of online, patient-evaluated diagnoses will reduce the validity of the diagnoses and reduce the generalizability to the clinical practice of trial results.

The use of a research-based clinical diagnostic assessment such as the Schedules for Clinical Assessment in Neuropsychiatry providing Diagnostic and Statistical Manual of Mental Disorders and ICD-10 diagnosis [49] or another systematic diagnostic assessment system should be prioritized. If patients are thoroughly and validly diagnosed and characterized before inclusion in the RCT, that is, by their treating doctors or psychologists, the clinical diagnoses may as well be used, depending on the aims and hypotheses of the RCT.

Smartphones allow for a vast amount of data to be collected online or automatically. This could be used as an advantage when conducting large RCTs. Digital solutions let researchers reach more participants and make it easier for patients to participate in clinical trials, not having to show up in the research settings for diagnostic assessment, baseline data collection, and outcome assessments. This possibly facilitates broader and more feasible inclusion of patients. Questionnaires conducted on the screen of a smartphone might vary compared with validated paper-based questionnaires. However, a Cochrane review from 2015 concludes that “apps might not affect data equivalence as long as the intended clinical application of the survey questionnaire, its intended frequency of administration and the setting in that it was validated remain unchanged” [50]. Telephonic interviews are different from clinical evaluations conducted by clinicians with possibly more reduced validity of diagnoses and sociodemographic and clinical data. Nevertheless, telephonic interviews (by trained lay interviewers or professionals) have been used in the research of depression and anxiety disorders with reasonable validity of diagnosis [51].

Modern electronic data collection should be used wisely with an awareness of possible changes in the quality of the data that the researchers obtain.

Interventions and Comparators

This systematic review showed that a placebo-controlled design was used to a limited extent. There is no clear definition of what a digital placebo treatment can or should contain, and defining a proper placebo group in a nonpharmacological RCTs is always difficult. The expectation for technical solutions themselves to be helpful resembles a reaction that is comparable to receiving a placebo pill and has been suggested to affect patients independent of active treatment [52]. The term digital placebo has been suggested [52]. Still, little is known about this issue; however, ongoing RCTs are investigating this subject further using a sham app as a comparator [53].

Clear descriptions of the content of the interventions and comparators used by researchers in future RCTs should be prioritized and made available to readers. If not mentioned in the primary publication, clear reference to the description should be made, such as in the study protocol, earlier publication, appendices, or publicly available versions of the intervention used. Furthermore, researchers could beneficially design the active intervention to fit into clinical practice and adapt to the clinical settings either as a stand-alone treatment or in combination with clinical treatment and support.

Outcome Measures and Power Analyses

A clear predefined research question, represented by a predefined primary outcome measure in a precisely well-defined patient population, is necessary. Selecting a predefined and relevant primary outcome measure is crucial [12]. Assessors should be blinded to outcome measures. End points of clinical relevance, that is independent of or blinded to researchers, such as admittance to a hospital or relapse or recurrence of illness (as in an ongoing trial [54]), should be prioritized compared with biased end points [12]. Such outcomes benefit from being critical for patients, relatives, and clinicians.

Power analyses should preferably be made before start the trial to ensure that the required sample is realistic and able to answer the primary research question—often represented by significant changes in the primary outcome.

When using clinician-based outcome measures, this should preferably be done by blinded trained clinical researchers. This is a difficult task because patients may give the researcher a hint on their allocation. Precautions to hinder this should be taken and described.

Although patient-evaluated outcome measures are appealing because of the ease use, we should be careful when interpreting findings based on unblinded patient-evaluated outcomes as there will be a risk of bias.

Clinicians and administrators may play important roles in implementing such treatment systems following trial findings if the results indicate a possible effect of the new treatment [55].

Possible outcomes could include measures of clinicians’ and caregivers’ attitudes toward the intervention if involved. Further data regarding the use of resources and economic costs related to the intervention could be included if available.

Publication of a Trial Protocol and Reporting Guidelines

Compilation and publication of a trial protocol or a thorough description on publicly available registration sites, such as, before analyzing data from the trial will increase the validity of the findings. Trial protocols and registrations should follow the standards for medical trial protocols [20] such as precise descriptions of inclusion and exclusion criteria, recruitment procedures, prioritized outcome measures including unblinded assessment, statistical power analyses, and a plan for analyses of participants versus nonparticipants. Furthermore, a thorough description of the used technology should be included [54,56,57]. Authors should publish deviations on especially the technical side of the protocol that were necessary, including valid arguments for the changes.

When reporting results, guidelines such as the CONSORT eHealth checklist [20] should be followed to increase transparency and for evaluation of findings.

Multidisciplinary Research

A close collaboration between information technology developers, clinicians and researchers is critical to ensure that the technology developed can answer the primary questions that are addressed and, vice versa, that the posed hypotheses match the available technology.

Authors should use the same transparency when working with the technology industry as researchers, doctors, and funders use with the medical industry. This systematic review shows that the authors’ declarations of conflicts of interest are frequently missing. The industry has a significant direct or indirect interest in developing app solutions and gathers information from patients as information in the digital economy involves major economic interests [58]. Any potential conflicts of interest should always be declared.

Technology in Randomized Controlled Trials

Smartphones represent a unique tool to measure adherence and fidelity in RCTs. However, there is no clear definition of how to measure or report adherence to smartphone-based treatment systems.

Smartphone solutions might assist in follow-up assessments and could advantageously use built-in features to ease the patients. Even when patients are “lost to follow-up,” assessments based on smartphone sensors could be collected, with patient permission, and might reflect behavioral changes for the user [59-61].

Prompts and reminders may comprise an important part of digital solutions to keep patients engaged in treatment.

These prompts, preferably, should be a part of the intended intervention that is investigated and not done only as part of the research project. The research team will potentially not be part of a real-world implementation of the technology and therefore give a false picture of adherence and effect of the intervention.

Information about technical skills and smartphone usage and habits might be valuable information and should be presented.

Reporting possible harms and adverse events are essential as the field of smartphone-based treatment is new, and little is known of potentially harmful effects such as worsening of symptoms or suicidal ideation. One trial has reported possible harms of using the intervention [27].

Potential harms and safety matters should be taken into consideration when designing and conducting the trial, as the example of the study by Stiles-Shields et al, [42] in which a safety protocol was used to standardize how researchers should react on suicidal ideations, expressed by participants either in the app or as part of the online questionnaires (where suicidal thoughts is a common item).

Updates, Revisions, and Adaptive Trials

Developing well-designed RCTs, including proper diagnostic procedures and thorough, robust, and blinded outcome measures, will inevitably be a time-consuming process. Still, it is crucial to test new smartphone-based treatment interventions on well-defined patient populations. Locking a digital intervention for several years is a challenge because technology would be expected to evolve and improve over time. In software engineering, updates and enhancements of the software are frequently released to fix bugs and errors and to improve usability, stability, robustness, and security. Moreover, the type of data collected from the phone via sensors and its processing by algorithms will also constantly change and adapt according to automatic machine learning models. Such learning systems will change based on new information and might be used increasingly in psychiatry as well as in many other fields [62]. Finally, the hardware and operating systems of the mobile phone regularly change, and the major vendors (Apple and Google) regularly release new hardware and operating systems that affect what an app can and cannot do.

Consequently, the concept of maintaining the digital intervention stable during a clinical trial is unrealistic, taking into consideration the long time span involved in conducting most RCTs.

To merge these opposing interests, we must accept these ongoing changes as a natural consequence of conducting research in the field, similar to surgeons who improve their skills or therapist who adapt and improve as they see more and more patients. Such updates should be reported systematically and should be thoroughly described so that it will be clear to readers which type of updates were made and what effect they might have had on trial results. In addition, the possible effects of add-ons could be presented and provided enough statistical power for subanalyses.

By allowing a more flexible approach to the RCT design, it would conform more to the so-called adaptive trial paradigm [17,63], and then RCTs might be able to answer questions regarding the effect of mHealth interventions without evaluating outdated technology [16,17].

It is necessary to accept and incorporate technological developments and changes in the design of a specific RCT in line with suggestions from similar but older scientific fields as telehealth in general [64].

Ethical Aspects

In a digital economy, data generated by apps and services constitute a valuable resource. Within psychiatry and other areas, such data can lead to discrimination of individuals and patient groups as a whole, and therefore, the handling and use of the data collected are of great importance. The value such information has to industry and other stakeholders is driving the many apps that are widely available without evidence of effects or possible harms. We need information obtained through well-designed and transparent trials to improve our knowledge of how these treatments can help patients and professionals.

Privacy, security, technology illiteracy, depersonalization of treatment systems, and technological paternalism are some of many possible ethical issues in this field [58].

These challenges apply both to the trial designs and the treatments being developed. The included trials hardly mentioned any such ethical issues.

Trials must consider and discuss possible ethical implications of the trial design and of the treatment itself.

Findings in a Scientific Context

The results from one of the included trials have been replicated using the same intervention, trial design, and outcome measures in a second trial only once [27,40]. However, this is necessary to increase evidence. Notably, numerous trial protocols within the field were identified during the search [54,65-69]. This indicates that the pace of the field is still increasing, and hopefully, future trials will provide more evidence within the field of smartphone-based treatment in patients with psychiatric disorders. This development is further shown in the large numbers of RCTs found in this review. Only 1 of the included trials in this review was published and included in a previous review from 2013 investigating the effects of smartphones in mental health [18]. With an increasing number of RCTs and hopefully increasing the quality of design and method, and a more uniform reporting of results, future meta-analyses of the effects of treatments will be possible.


This study has various limitations: not all parts of the review process and data collection were double checked, no protocol of study method was published online beforehand, and no meta-analyses or statistical analyses were performed on the included data. We did not use any scoring systems for the assessments of quality and risk of bias in the included studies, leaving the assessment to the reader.


This first systematic review on the design, conduct, and methodological challenges of RCTs investigating the effect of smartphone-based treatment in patients with a psychiatric diagnosis suggests that there is a rapidly increasing interest for this type of treatment. Although an increasing number of trials tested new smartphone-based treatments, the trial designs and reporting were of low quality compared with more classic medical RCTs, and heterogeneity and methodological issues in individual trials limit the evidence.

Smartphone-based treatment interventions imply new challenges and opportunities, but we, as researchers, should consider strict methodological efforts when designing, conducting, and reporting such trials as in the rest of the field of medicine. Future trials employing strict methodology, including detailed description regarding patient recruitment, pre- and well-defined, prioritized outcome measures, information regarding technical updates and down periods, and statements on potential conflicts of interest are warranted. Research groups without trial experience should seek out information on how to conduct RCTs with high a methodological standard to ensure a high level of quality in the research. Finally, close collaborations between professions and specialties are needed in this complex branch of science.


MLT’s work is supported by the Innovation Foundation (5164-00001B).

Conflicts of Interest

All authors are currently working on the RADMIS project, which is the successor of the MONARCA system. LVK has been a consultant for Sunovion in the past 3 years. JEB is a cofounder and chief scientific officer in MONSENSO, a technology company developing and selling app-based solutions for psychiatric diseases. The other authors have no conflicts to declare.

Multimedia Appendix 1

Data extraction template.

PDF File (Adobe PDF File), 116 KB

  1. GBD 2017 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. Lancet 2018 Nov 10;392(10159):1789-1858 [FREE Full text] [CrossRef] [Medline]
  2. Trautmann S, Rehm J, Wittchen H. The economic costs of mental disorders: do our societies react appropriately to the burden of mental disorders? EMBO Rep 2016 Sep;17(9):1245-1249 [FREE Full text] [CrossRef] [Medline]
  3. Wittchen H, Jacobi F. Size and burden of mental disorders in Europe--a critical review and appraisal of 27 studies. Eur Neuropsychopharmacol 2005 Aug;15(4):357-376. [CrossRef] [Medline]
  4. World Health Organization. Fact Sheet on Mental Disorders   URL: [accessed 2018-12-13]
  5. World Health Organization. mHealth: New Horizons for Health Through Mobile Technologies. Geneva, Switzerland: World Health Organization; 2011.
  6. Statista. Number of Smartphone Users Worldwide From 2016 to 2021 (in Billions)   URL: [accessed 2018-12-13]
  7. Netimperative. 2017. Denmark ‘Has Highest Smartphone Penetration Rate in the World’   URL: [accessed 2018-12-13]
  8. Dogan E, Sander C, Wagner X, Hegerl U, Kohls E. Smartphone-based monitoring of objective and subjective data in affective disorders: where are we and where are we going? Systematic review. J Med Internet Res 2017 Jul 24;19(7):e262 [FREE Full text] [CrossRef] [Medline]
  9. Faurholt-Jepsen M, Munkholm K, Frost M, Bardram JE, Kessing LV. Electronic self-monitoring of mood using IT platforms in adult patients with bipolar disorder: a systematic review of the validity and evidence. BMC Psychiatry 2016 Jan 15;16:7 [FREE Full text] [CrossRef] [Medline]
  10. Wang K, Varma DS, Prosperi M. A systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disorders. J Psychiatr Res 2018 Dec;107:73-78. [CrossRef] [Medline]
  11. Anthes E. Mental health: there's an app for that. Nature 2016 Apr 7;532(7597):20-23. [CrossRef] [Medline]
  12. Schulz KF, Grimes DA. Generation of allocation sequences in randomised trials: chance, not choice. Lancet 2002 Feb 9;359(9305):515-519. [CrossRef] [Medline]
  13. Strand LB, Clarke P, Graves N, Barnett AG. Time to publication for publicly funded clinical trials in Australia: an observational study. BMJ Open 2017 Mar 22;7(3):e012212 [FREE Full text] [CrossRef] [Medline]
  14. Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med 2013 May 10;2(1):10 [FREE Full text] [CrossRef] [Medline]
  15. Mohr DC, Schueller SM, Riley WT, Brown CH, Cuijpers P, Duan N, et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res 2015 Jul 8;17(7):e166 [FREE Full text] [CrossRef] [Medline]
  16. Baker TB, Gustafson DH, Shah D. How can research keep up with eHealth? Ten strategies for increasing the timeliness and usefulness of eHealth research. J Med Internet Res 2014 Feb 19;16(2):e36 [FREE Full text] [CrossRef] [Medline]
  17. Law LM, Wason JM. Design of telehealth trials--introducing adaptive approaches. Int J Med Inform 2014 Dec;83(12):870-880 [FREE Full text] [CrossRef] [Medline]
  18. Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013 Nov 15;15(11):e247 [FREE Full text] [CrossRef] [Medline]
  19. Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, et al. The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry 2017 Oct;16(3):287-298 [FREE Full text] [CrossRef] [Medline]
  20. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions. J Med Internet Res 2011 Dec 31;13(4):e126 [FREE Full text] [CrossRef] [Medline]
  21. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J 2009 Jul 21;339:b2535 [FREE Full text] [CrossRef] [Medline]
  22. Watts S, Mackenzie A, Thomas C, Griskaitis A, Mewton L, Williams A, et al. CBT for depression: a pilot RCT comparing mobile phone vs computer. BMC Psychiatry 2013 Feb 7;13:49 [FREE Full text] [CrossRef] [Medline]
  23. Dagöö J, Asplund RP, Bsenko HA, Hjerling S, Holmberg A, Westh S, et al. Cognitive behavior therapy versus interpersonal psychotherapy for social anxiety disorder delivered via smartphone and computer: a randomized controlled trial. J Anxiety Disord 2014 May;28(4):410-417. [CrossRef] [Medline]
  24. Gustafson DH, McTavish FM, Chih M, Atwood AK, Johnson RA, Boyle MG, et al. A smartphone application to support recovery from alcoholism: a randomized clinical trial. JAMA Psychiatry 2014 May;71(5):566-572 [FREE Full text] [CrossRef] [Medline]
  25. Ly KH, Trüschel A, Jarl L, Magnusson S, Windahl T, Johansson R, et al. Behavioural activation versus mindfulness-based guided self-help treatment administered through a smartphone application: a randomised controlled trial. BMJ Open 2014 Jan 9;4(1):e003440 [FREE Full text] [CrossRef] [Medline]
  26. Depp CA, Ceglowski J, Wang VC, Yaghouti F, Mausbach BT, Thompson WK, et al. Augmenting psychoeducation with a mobile intervention for bipolar disorder: a randomized controlled trial. J Affect Disord 2015 Mar 15;174:23-30 [FREE Full text] [CrossRef] [Medline]
  27. Faurholt-Jepsen M, Frost M, Ritz C, Christensen EM, Jacoby AS, Mikkelsen RL, et al. Daily electronic self-monitoring in bipolar disorder using smartphones - the MONARCA I trial: a randomized, placebo-controlled, single-blind, parallel group trial. Psychol Med 2015 Oct;45(13):2691-2704. [CrossRef] [Medline]
  28. Ly KH, Topooco N, Cederlund H, Wallin A, Bergström J, Molander O, et al. Smartphone-supported versus full behavioural activation for depression: a randomised controlled trial. PLoS One 2015;10(5):e0126559 [FREE Full text] [CrossRef] [Medline]
  29. Moëll B, Kollberg L, Nasri B, Lindefors N, Kaldo V. Living SMART — a randomized controlled trial of a guided online course teaching adults with ADHD or sub-clinical ADHD to use smartphones to structure their everyday life. Internet Interv 2015 Mar;2(1):24-31. [CrossRef]
  30. Ivanova E, Lindner P, Ly KH, Dahlin M, Vernmark K, Andersson G, et al. Guided and unguided acceptance and commitment therapy for social anxiety disorder and/or panic disorder provided via the internet and a smartphone application: a randomized controlled trial. J Anxiety Disord 2016 Dec;44:27-35. [CrossRef] [Medline]
  31. Hildebrandt T, Michaelides A, Mackinnon D, Greif R, DeBar L, Sysko R. Randomized controlled trial comparing smartphone assisted versus traditional guided self-help for adults with binge eating. Int J Eat Disord 2017 Nov;50(11):1313-1322 [FREE Full text] [CrossRef] [Medline]
  32. Mantani A, Kato T, Furukawa TA, Horikoshi M, Imai H, Hiroe T, et al. Smartphone cognitive behavioral therapy as an adjunct to pharmacotherapy for refractory depression: randomized controlled trial. J Med Internet Res 2017 Nov 3;19(11):e373 [FREE Full text] [CrossRef] [Medline]
  33. Ben-Zeev D, Brian RM, Jonathan G, Razzano L, Pashka N, Carpenter-Song E, et al. Mobile Health (mHealth) versus clinic-based group intervention for people with serious mental illness: a randomized controlled trial. Psychiatr Serv 2018 Sep 1;69(9):978-985. [CrossRef] [Medline]
  34. Boettcher J, Magnusson K, Marklund A, Berglund E, Blomdahl R, Braun U, et al. Adding a smartphone app to internet-based self-help for social anxiety: a randomized controlled trial. Comput Hum Behav 2018 Oct;87(6):98-108. [CrossRef]
  35. Bucci S, Barrowclough C, Ainsworth J, Machin M, Morris R, Berry K, et al. Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophr Bull 2018 Aug 20;44(5):1070-1080 [FREE Full text] [CrossRef] [Medline]
  36. Hur J, Kim B, Park D, Choi S. A scenario-based cognitive behavioral therapy mobile app to reduce dysfunctional beliefs in individuals with depression: a randomized controlled trial. Telemed J E Health 2018 Sep;24(9):710-716. [CrossRef] [Medline]
  37. Liang D, Han H, Du J, Zhao M, Hser Y. A pilot study of a smartphone application supporting recovery from drug addiction. J Subst Abuse Treat 2018 May;88:51-58 [FREE Full text] [CrossRef] [Medline]
  38. Schlosser DA, Campellone TR, Truong B, Etter K, Vergani S, Komaiko K, et al. Efficacy of PRIME, a mobile app intervention designed to improve motivation in young people with schizophrenia. Schizophr Bull 2018 Aug 20;44(5):1010-1020 [FREE Full text] [CrossRef] [Medline]
  39. Stolz T, Schulz A, Krieger T, Vincent A, Urech A, Moser C, et al. A mobile app for social anxiety disorder: a three-arm randomized controlled trial comparing mobile and PC-based guided self-help interventions. J Consult Clin Psychol 2018 Jun;86(6):493-504. [CrossRef] [Medline]
  40. Faurholt-Jepsen M, Frost M, Christensen EM, Bardram JE, Vinberg M, Kessing LV. The effect of smartphone-based monitoring on illness activity in bipolar disorder: the MONARCA II randomized controlled single-blinded trial. Psychol Med 2019 Apr 4:- (epub ahead of print). [CrossRef] [Medline]
  41. Krzystanek M, Borkowski M, Skałacka K, Krysta K. A telemedicine platform to improve clinical parameters in paranoid schizophrenia patients: results of a one-year randomized study. Schizophr Res 2019 Feb;204:389-396. [CrossRef] [Medline]
  42. Stiles-Shields C, Montague E, Kwasny MJ, Mohr DC. Behavioral and cognitive intervention strategies delivered via coached apps for depression: pilot trial. Psychol Serv 2019 May;16(2):233-238. [CrossRef] [Medline]
  43. Teng M, Hou Y, Chang S, Cheng H. Home-delivered attention bias modification training via smartphone to improve attention control in sub-clinical generalized anxiety disorder: a randomized, controlled multi-session experiment. J Affect Disord 2019 Mar 1;246:444-451. [CrossRef] [Medline]
  44. Enock PM, Hofmann SG, McNally RJ. Attention bias modification training via smartphone to reduce social anxiety: a randomized, controlled multi-session experiment. Cogn Ther Res 2014 Mar 4;38(2):200-216. [CrossRef]
  45. Roepke AM, Jaffee SR, Riffle OM, McGonigal J, Broome R, Maxwell B. Randomized controlled trial of SuperBetter, a smartphone-based/internet-based self-help tool to reduce depressive symptoms. Games Health J 2015 Jun;4(3):235-246. [CrossRef] [Medline]
  46. Miner A, Kuhn E, Hoffman JE, Owen JE, Ruzek JI, Taylor CB. Feasibility, acceptability, and potential efficacy of the PTSD Coach app: a pilot randomized controlled trial with community trauma survivors. Psychol Trauma 2016 May;8(3):384-392. [CrossRef] [Medline]
  47. Possemato K, Kuhn E, Johnson E, Hoffman JE, Owen JE, Kanuri N, et al. Using PTSD coach in primary care with and without clinician support: a pilot randomized controlled trial. Gen Hosp Psychiatry 2016;38:94-98. [CrossRef] [Medline]
  48. Kuhn E, Kanuri N, Hoffman JE, Garvert DW, Ruzek JI, Taylor CB. A randomized controlled trial of a smartphone app for posttraumatic stress disorder symptoms. J Consult Clin Psychol 2017 Mar;85(3):267-273. [CrossRef] [Medline]
  49. Wing JK, Babor T, Brugha T, Burke J, Cooper JE, Giel R, et al. SCAN. Schedules for clinical assessment in neuropsychiatry. Arch Gen Psychiatry 1990 Jun;47(6):589-593. [CrossRef] [Medline]
  50. Belisario JS, Jamsek J, Huckvale K, O'Donoghue J, Morrison CP, Car J. Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev 2015 Jul 27(7):MR000042. [CrossRef] [Medline]
  51. Muskens EM, Lucassen P, Groenleer W, van Weel C, Oude Voshaar R, Speckens A. Psychiatric diagnosis by telephone: is it an opportunity? Soc Psychiatry Psychiatr Epidemiol 2014 Oct;49(10):1677-1689. [CrossRef] [Medline]
  52. Torous J, Firth J. The digital placebo effect: mobile mental health meets clinical psychiatry. Lancet Psychiatry 2016 Feb;3(2):100-102. [CrossRef] [Medline]
  53. Giosan C, Cobeanu O, Mogoaşe C, Szentagotai A, Mureşan V, Boian R. Reducing depressive symptomatology with a smartphone app: study protocol for a randomized, placebo-controlled trial. Trials 2017 May 12;18(1):215 [FREE Full text] [CrossRef] [Medline]
  54. Faurholt-Jepsen M, Frost M, Martiny K, Tuxen N, Rosenberg N, Busk J, et al. Reducing the rate and duration of re-admissions among patients with unipolar disorder and bipolar disorder using smartphone-based monitoring and treatment - the RADMIS trials: study protocol for two randomized controlled trials. Trials 2017 Jun 15;18(1):277 [FREE Full text] [CrossRef] [Medline]
  55. Kerst A, Zielasek J, Gaebel W. Smartphone applications for depression: a systematic literature review and a survey of health care professionals' attitudes towards their use in clinical practice. Eur Arch Psychiatry Clin Neurosci 2019 Jan 03:- (epub ahead of print). [CrossRef] [Medline]
  56. Faurholt-Jepsen M, Vinberg M, Christensen EM, Frost M, Bardram J, Kessing LV. Daily electronic self-monitoring of subjective and objective symptoms in bipolar disorder--the MONARCA trial protocol (MONitoring, treAtment and pRediCtion of bipolAr disorder episodes): a randomised controlled single-blind trial. BMJ Open 2013;3(7):- [FREE Full text] [CrossRef] [Medline]
  57. Faurholt-Jepsen M, Vinberg M, Frost M, Christensen EM, Bardram J, Kessing LV. Daily electronic monitoring of subjective and objective measures of illness activity in bipolar disorder using smartphones--the MONARCA II trial protocol: a randomized controlled single-blind parallel-group trial. BMC Psychiatry 2014 Nov 25;14:309 [FREE Full text] [CrossRef] [Medline]
  58. Bauer M, Glenn T, Monteith S, Bauer R, Whybrow PC, Geddes J. Ethical perspectives on recommending digital technology for patients with mental illness. Int J Bipolar Disord 2017 Dec;5(1):6 [FREE Full text] [CrossRef] [Medline]
  59. Faurholt-Jepsen M, Bauer M, Kessing LV. Smartphone-based objective monitoring in bipolar disorder: status and considerations. Int J Bipolar Disord 2018 Jan 23;6(1):6 [FREE Full text] [CrossRef] [Medline]
  60. Faurholt-Jepsen M, Frost M, Vinberg M, Christensen EM, Bardram JE, Kessing LV. Smartphone data as objective measures of bipolar disorder symptoms. Psychiatry Res 2014 Jun 30;217(1-2):124-127. [CrossRef] [Medline]
  61. Faurholt-Jepsen M, Brage S, Vinberg M, Jensen HM, Christensen EM, Knorr U, et al. Electronic monitoring of psychomotor activity as a supplementary objective measure of depression severity. Nord J Psychiatry 2015 Feb;69(2):118-125. [CrossRef] [Medline]
  62. Bzdok D, Meyer-Lindenberg A. Machine learning for precision psychiatry: opportunities and challenges. Biol Psychiatry Cogn Neurosci Neuroimaging 2018 Mar;3(3):223-230. [CrossRef] [Medline]
  63. Park JJ, Thorlund K, Mills EJ. Critical concepts in adaptive clinical trials. Clin Epidemiol 2018;10:343-351 [FREE Full text] [CrossRef] [Medline]
  64. Ekeland AG, Bowes A, Flottorp S. Methodologies for assessing telemedicine: a systematic review of reviews. Int J Med Inform 2012 Jan;81(1):1-11. [CrossRef] [Medline]
  65. Mühlbauer E, Bauer M, Ebner-Priemer U, Ritter P, Hill H, Beier F, et al. Effectiveness of smartphone-based ambulatory assessment (SBAA-BD) including a predicting system for upcoming episodes in the long-term treatment of patients with bipolar disorders: study protocol for a randomized controlled single-blind trial. BMC Psychiatry 2018 Oct 26;18(1):349 [FREE Full text] [CrossRef] [Medline]
  66. Anastasiadou D, Lupiañez-Villanueva F, Faulí C, Arcal Cunillera J, Serrano-Troncoso E. Cost-effectiveness of the mobile application TCApp combined with face-to-face CBT treatment compared to face-to-face CBT treatment alone for patients with an eating disorder: study protocol of a multi-centre randomised controlled trial. BMC Psychiatry 2018 May 2;18(1):118 [FREE Full text] [CrossRef] [Medline]
  67. Scott CK, Dennis ML, Gustafson DH. Using smartphones to decrease substance use via self-monitoring and recovery support: study protocol for a randomized control trial. Trials 2017 Aug 10;18(1):374 [FREE Full text] [CrossRef] [Medline]
  68. Kolar DR, Hammerle F, Jenetzky E, Huss M. Smartphone-enhanced low-threshold intervention for adolescents with anorexia nervosa (SELTIAN) waiting for outpatient psychotherapy: study protocol of a randomised controlled trial. BMJ Open 2017 Oct 22;7(10):e018049 [FREE Full text] [CrossRef] [Medline]
  69. Kemmeren LL, van Schaik DJF, Riper H, Kleiboer AM, Bosmans JE, Smit JH. Effectiveness of blended depression treatment for adults in specialised mental healthcare: study protocol for a randomised controlled trial. BMC Psychiatry 2016 Apr 21;16:113 [FREE Full text] [CrossRef] [Medline]

BDI-II: Beck Depression Inventory
CBT: cognitive behavioral therapy
CES-D: Center for Epidemiologic Studies Depression Scale
CONSORT: Consolidated Standards of Reporting Trials
DSM: Diagnostic and Statistical Manual of Mental Disorders
eHealth: electronic health
ICD-10: International Statistical Classification of Diseases and Related Health Problems-10
K-10: Kessler Psychological Distress Scale
LSAS-SR: Liebowitz Social Anxiety Scale—self-reported
MADRS: Montgomery–Åsberg Depression Rating Scale
mHealth: mobile health
MINI: Mini International Neuropsychiatric Interview
NI: noninferiority
PCL-C: Post-Traumatic Checklist—Civilian
PCL-S: Post-Traumatic Checklist Scale
PHQ-9: Patient Health Questionnaire-9
PTSD: posttraumatic stress disorder
RCT: randomized controlled trial
SCAN: Schedules for Clinical Assessment in Neuropsychiatry
SCID: Structured Clinical Interview.
SIAS: Social Interaction Anxiety Scale
TAU: Treatment as usual
YMRS: Young Mania Rating Scale

Edited by G Eysenbach; submitted 04.07.19; peer-reviewed by C Simons, D Ben-Zeev; comments to author 23.07.19; revised version received 28.08.19; accepted 04.09.19; published 27.10.19


©Morten Lindbjerg Tønning, Lars Vedel Kessing, Jakob Eivind Bardram, Maria Faurholt-Jepsen. Originally published in the Journal of Medical Internet Research (, 27.10.2019.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.