Methodological Challenges in Randomized Controlled Trials on Smartphone-Based Treatment in Psychiatry: Systematic Review

Introduction

Background

Psychiatric disorders represent a major burden of disease worldwide with a significant impact on the quality of life, socioeconomic factors, and life expectancy [1]. In 2010, the worldwide expenses because of mental illness were estimated to be between US $2.5 trillion and US $8.5 trillion [2]. Across European countries, 27% of the adult population suffers from at least one psychiatric disorder [3]. At the same time, there is a gap between the need for treatment and the number of patients receiving treatment. The number of patients who do not receive treatment for their disorder is 35% to 50% in high-income countries and 76% to 85% and in low- and middle-income countries [4].

In 2011, the World Health Organization stated that “the use of mobile and wireless technologies to support the achievement of health objectives (mHealth) has the potential to transform the face of health service delivery across the globe” [5]. The number of smartphone users exceeded 2.5 billion people in 2018 [6], and in high-income countries, 80% of the population own and use a smartphone [7].

Smartphones are a promising tool in the field of psychiatry. They are widely used and always at hand, allowing for delivery of treatment to patients in real-time and naturalistic settings, and can augment already available treatments. At the same time, smartphones contain several sensors and technologies enabling patients, researchers, and clinicians to access information about physical and social activities [8].

Mobile health (mHealth) and especially smartphone-based technology and solutions are developing at an enormous speed, driven mainly by software and computer scientists and private companies. Thus, most available apps have not been scientifically investigated, and the validity, treatment effect, and safety have been sparingly investigated [9,10]. Nevertheless, hundreds of apps claiming to help or monitor psychiatric disorders are already available in app stores [11].

Evidence, Randomized Controlled Trials, and Interdisciplinary Research

Randomized controlled trials (RCTs) represent the methodological golden standard of excellence in medical research for the investigation of possible positive and negative effects of treatment interventions [12]. The importance of RCTs in medical science is mainly because of their ability to eliminate confounding of known and unknown nature. If properly designed and conducted, RCTs are especially useful for examination of small or moderate positive and negative effects [12].

The design and conduct of RCTs within smartphone-based treatment interventions and other electronic treatments should follow the RCT standards within medicine, while also taking into account particular challenges with electronic effect research. In RCTs, in general, there is a significant time gap from design and trial initiation to the publication of results. A recent Australian study of publicly funded clinical trials showed a median time of 7.1 years (95% CI 6.3-7.6 years) from funding to the main article on trial results being published [13]. This time gap is particularly problematic when testing smartphone-based treatment interventions as the technological development taking place within the timespan of an RCT is enormous. The technology tested is at risk of being outdated when results are published and before being taken into clinical usage [14]. Furthermore, the locked nature of treatment interventions tested in RCTs contrasts with the constantly evolving and iterative nature of app solutions [15].

Previous Reviews

Other groups have previously suggested changes to the development process, to speed up the process from idea to publication [14,16,17]. A previous review from 2013 [18] identified effect studies of smartphone-based treatments within psychiatry (including stress). This review was not limited to include RCTs, and thus, it also included pre- to posttest studies. Only 8 studies were identified despite an extensive search and broad inclusion criteria.

Symptom-specific reviews within branches of psychiatry have been done as well. As an example, Firth et al [19] found studies regarding depressive symptoms and conducted a meta-analysis on the possible effects of smartphone-based treatments on these symptoms. A total of 18 RCTs were identified; however, often, a psychiatric diagnosis was not present as the focus was on depressive symptoms. This study focused on RCTs only. The focus was on research methods and design in relation to smartphone-based treatment interventions within psychiatry. This study was limited to RCTs providing information on diagnostic measures to ensure that participants suffers from a psychiatric diagnosis, excluding trials focusing on symptoms in healthy populations.

Aims

This systematic review aimed to identify and describe all available RCTs using smartphones as (part of) a treatment intervention conducted in the field of psychiatry using sufficient diagnostic measures. Furthermore, it aimed to describe the methodology of these individual trials and discuss methodological challenges related to designing and conducting RCTs within smartphone-based treatment in psychiatric disorders using the Consolidated Standards of Reporting Trials (CONSORT) electronic health (eHealth) checklist [20] as a reference and to provide recommendations for future trials within the area.

Methods

Methods of the review and eligibility criteria were established in advance by 3 of the authors (MLT, MFJ, and LVK). Minor modifications such as adding further information to be extracted were made to the review protocol during the review process.

Trial Selection

Original trials reporting on smartphone-based treatment interventions investigated in RCTs, including adult patients with psychiatric disorders, were eligible for review. In addition, peer-reviewed articles, posters, and conference abstracts were eligible for review. If multiple articles reported on overlapping populations, the article presenting the largest population was included. No restrictions regarding sample size were applied.

The exclusion criteria were as follows: (1) children younger than 18 years; (2) psychiatric symptoms as part of somatic disorders (ie, preoperation anxiety or depressive symptoms in terminal cancer patients); (3) trials concerning stress, cigarette smoking, low intelligence quotient (IQ), and isolated sleep problems without psychiatric disorders; (4) trials with individuals who self-identified as having a psychiatric diagnosis but without diagnostic reassurance; (5) trials reporting on symptoms without diagnoses (ie, depressive symptoms or alcohol usage among college students); (6) trials using internet therapy without an active smartphone-based component (ie, if the Web page was accessible from a smartphone browser); (7) trials using only cell phones in traditional ways with text messages and phone calls (not using smartphone-based features); (8) trials using smartphones as a screen for virtual-reality setups as the primary treatment component; and (9) trials not available in English.

Information Source, Trial Selection, and Data Extraction

We conducted a systematic search covering PubMed, PsycINFO, and EMBASE on April 23, 2018, and it was last updated on August 12, 2019. Only articles from 2008 onwards were included (the time of the release of the first smartphone). References from articles and other reviews were also examined, but they did not result in any additional trials to include. The trial selection was conducted by 2 researchers (MFJ and MLT), and articles with doubt about the relevance were discussed between the 2 of them. Full-text articles for possible relevant trials were obtained if the abstract and title did not supply sufficient information. The search strategy included (1) psychiatric disorder as a broad term and for specific diseases AND (2) smartphone or app AND (3) RCT. A wide variety of text words were used to include trials published within the last 6 months that had not yet been indexed with Medical Subject Headings terms. Search strategy in PubMed was as follows:

(((Smartphone[MeSH terms] OR mobile application[MeSH Terms] OR smartphone OR mobile application OR smart phone OR mobile phone OR app OR apps OR cell phone OR Iphone* OR IOS OR Android phone OR smartphones OR mobile applications OR smart phones OR mobile phones OR cell phones)) AND (((((((((mental disorder[MeSH Terms]) OR (mental disorder OR mental disorders OR mental disease OR mental diseases OR mental diagnose OR psychiatric disease OR psychiatric diseases OR psychiatric disorders OR psychiatric disorder OR psychiatric diagnose)) OR ((drug OR substance OR prescription drug OR alcohol OR narcotic OR heroin OR amphetamine OR cocaine OR marijuana OR opioid OR morphine OR phencyclidine) AND (abuse OR dependence OR addiction))) OR (feeding disorder OR feeding disorders OR eating disorders OR eating disorder OR anorexia OR bulimia OR binge eating)) OR (autism OR autistic OR Asperger disease OR Aspergers disease) OR Asperger disorder OR Aspergers disorder OR ADHD OR attention deficit disorder OR ADD OR attention deficit hyperactivity disorder)) OR (personality disorder OR personality disorders OR obsessive-compulsive personality OR compulsive personality OR obsessive personality OR psychopath OR sociopathic OR antisocial OR passive-dependent personality OR dyssocial OR schizoid OR schizotypal)) OR (schizophrenia OR psychoses OR psychosis OR psychotic OR paranoid OR schizoaffective OR schizophreniform OR delusional)) OR (major depressive disorder OR unipolar depression OR unipolar disorder OR depressive syndrome OR endogenous depression OR neurotic depression OR melancholia OR cyclothymic OR dysthymic OR mood disorder OR mood disorders OR affective disorder OR affective disorders OR bipolar OR manic-depressive OR mania OR manic) OR (anxiety OR anxieties OR panic disorder OR agoraphobia OR obsessive disorder OR compulsive disorder OR obsessive-compulsive disorder OR phobic disorder OR phobic disorders OR PTSD OR posttraumatic stress disorder OR post-traumatic stress disorder OR post traumatic stress disorder))) AND ((randomized controlled trial[MeSH Terms]) OR (randomized controlled trial OR randomised controlled trial OR randomised OR randomized OR RCT OR randomized clinical trial OR randomiced clinical trial OR randomized clinical trial OR randomized controlled clinical trial OR randomised controlled clinical trial))

Data were extracted by using the CONSORT eHealth checklist as a template for data extraction [20]. The following data were extracted:

• Author, year, country, trial design, trial registration, protocol availability, patients’ age and gender, sample size and use of power calculations, length of treatment intervention, and follow-up period.
• Recruitment and diagnostics procedures of patients, recruitment length, outcome measures, well-defined hierarchy of outcome measures, and collection of outcome data.
• Description of treatment intervention and comparator, use of blended treatment and standard treatments, affiliation with industry and technology descriptions.
• Title according to CONSORT recommendations; use of prompts, platform choice, and possible lent-out of smartphones; economic compensation; use of placebo smartphones; methodological information regarding randomization and blinding procedures; information about the statistical approach to technical updates and whether updates and technical crashes or down periods were reported; possible harms; adherence to the smartphone system; and baseline data on patients’ technological skills and funding information.

A data extraction template is provided in the supplementary material (Multimedia Appendix 1). The extracted data are presented in 4 tables. Data were initially retrieved by one author (MLT) and subsequently and independently by another author (MFJ). Any disagreements were solved between MLT and MFJ.

Three tables describe the trials individually, with various focus. The fourth uses the CONSORT eHealth checklist [20] as a template to summarize the relevant findings according to these in a systematic way and hereby partly summarizes the relevant findings from previous tables, but it also includes other new relevant information. The tables are presented in relevant sections in the result-section.

Results

Trial Selection

Figure 1 presents a flow diagram represented according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines [21], showing the results of the literature search and selection of trials. The initial literature search in PubMed, EMBASE, and PsycINFO on April 23, 2018, resulted in a total of 1490 articles. Of these, 380 duplicates were removed, resulting in 1110 articles. Furthermore, 833 articles were excluded based on the title and year of publication with the main reasons for exclusion being as follows: not concerning psychiatric disorders (eg, HIV, Alzheimer disease, and obesity), not reporting on results from RCTs, and published before 2008. This led to a total of 277 remaining articles, from which abstracts were examined.

‎View this figure

Out of these 277 articles, 239 were excluded based on abstracts, with majority of reasons being as follows: trial protocols, text message or other nonsmartphone technology, publications on same data (the publication with the most data was included), measuring on symptoms—not diagnosis, no or insufficient diagnostic reassurance, participants younger than 18 years, topics concerning low IQ, smoking, stress, or isolated sleep problems.

This led to 38 articles that were printed for full-text reading. Among them, 19 articles were subsequently excluded because of the following: reporting symptoms, not disorders (n=7); no or insufficient diagnostic reassurance (n=6); only abstract in English (n=1); insomnia (n=1); no outcome effect measures (n=1); and not using smartphone as part of treatment (ie, only using smartphones to reach the patient or allowing patients to answer emails and questionnaires on smartphone; n=3). References from articles and other reviews were also examined, but they did not result in any additional trials to include, giving a total of 19 eligible articles from the initial search.

The search was updated on August 12, 2019, resulting in 8 new articles. Thus, a total of 27 unique trials [22-48] were identified and included in this review, including 5 with diagnostic estimates based on questionnaires using relevant cutoff scores [44-48]. Included trials are described individually in the tables.

Trial Characteristics

Across all included trials, a total of 3,312 patients were included. The included trials represented a range of psychiatric disorders. Numbers in parentheses represent relevant chapter and coding according to International Statistical Classification of Diseases and Related Health Problems-10 (ICD-10): 2 on drug or alcohol misuse (F10-F19) [24,37], 3 on psychosis (F20-29) [35,38,41], 10 on affective disorders (F30-39) [22,25-28,32,36,40,42,45] (comprising 3 on bipolar disorder and 7 on unipolar depressive disorder), 9 on anxiety and posttraumatic stress disorder posttraumatic stress disorder (F40-F48) [23,30,34,39,43,44,46-48], 1 on eating disorders (F50-F59) [31], and 1 on attention-deficit disorder (F90-F98) [29]. One trial included patients with severe mental disorders across ICD-10 diagnoses [33].

Most trials had an overrepresentation of females, except trials concerning veterans, schizophrenia, or substance or alcohol abuse. Patients’ ages ranged from 18 to 73 years. In the majority of the trials, the average age was 40 years.

Trial Design and Reporting

The number of RCTs testing smartphone solutions in patients with a psychiatric disorder increased with time, especially from 2018 onward. Some trials [24,27,29-33,35,40] were leaning toward traditional RCT designs conducted in clinical settings and using the CONSORT checklist for reporting and designing trials [20]. Some other trials did not provide clear information regarding these issues, leaving the reader with lacking information on the design and conduct of the trial.

Overall, 11 trials had a sample size above 100 patients [24,30,32-34,39-41,44,45,48]. One trial [40] had been repeated using the same intervention, trial design, and outcome measures as a previous trial [27].

A total of 14 articles included information about trial registration, such as the clinicaltrials.gov [22,24-27,29-33,35,39-41]. For the remaining trials (including the study by Krzystanek et al [41], where we could not find any information on the registration information provided) it was not possible to see whether the primary outcome measure was predefined in the primary hypothesis leading to the trial design. In addition, 5 trials referred to a published trial protocol in the article [27,30,32,35,40]. One additional article attached the study protocol in the supplementary section when published [28].

Information about the individual trial designs, registration, sample characteristics, and length of intervention and follow-up is presented in Table 1.

Settings and Diagnostic Procedures

A total of 8 trials were conducted in traditional clinical settings, with patients referred by clinical staff or contacted in their treatment clinic [24,27,32,33,35,37,40,41]. Others used remote designs where patients who self-identified as having a psychiatric disorder applied for participation [22,23,25,26,28,30,34,38,39,42-44,46,48]. None of the trials compared participants with nonparticipants. All but 2 trials [43,44] presented a flowchart of eligible subjects with varying details on reasons not to participate. Trials with open recruitment (eg, internet forums, Web pages, Facebook, and advertising) [22,23,25,26,28-30,34,36,38,39,42-46,48] only had information on people who signed up. Completion rates were reported very differently and varied from 163 patients out of 164 (99.4% [163/164]) completing the primary outcome [32] to 74 out of 283 patients (26.1% [74/283]) [45]. In addition, 12 trials compensated for participation in trial assessments with money or gift cards [26,33-38,42,43,46-48].

Validation and certainty of diagnosis varied substantially, and often, a pragmatic setup was used, leaving the validity of the obtained diagnosis with uncertainty.

A total of 5 trials based their diagnosis solely on clinical-based information [24,33,35,37,41]. Furthermore, 15 trials used research-based diagnoses without clinical information, using either questionnaires [44-46,48], phone calls [22,23,25,28,34,42], video interviews [38], or personal interviews [31,36,39,43] to validate the self-reported diagnoses. In addition, 7 trials included patients with a clinical diagnosis that was subsequently assessed and validated by the researchers using questionnaires [47], phone calls [29,30], or personal interviews [26,27,32,40].

Hypotheses and Use of Predefined Hierarchical Outcome Measures

Overall, 18 trials included clearly described hypotheses [22,24-28,30-32,34,36,38-40,44,45,47,48], and 8 trials [22,37,41-44,46,47] did not distinguish between primary and secondary outcome measures in the article and had no hierarchy of outcomes.

Outcome Measures

A total of 8 trials included objective or observer-based primary outcome measures: One trial tested the levels of drug use in the urine as a specific primary outcome measure [37]. Another trial used objective measurements of feasibility, use, and attrition as the primary outcome [35]. One trial tested a specific task of motivated behavior [38]. In addition, 4 trials used scores on clinical ratings as the primary outcome measure assessing the level of depression and mania [26,27,40], levels of bulimic episodes [31], or psychotic symptoms [41]. The remaining trials used patient-reported outcome measures.

Within trials of similar diagnoses, authors used different scales and measures for the primary outcome measure. For instance, in measuring depression scores, the following outcome measures were used as the primary outcome: Patient Health Questionnaire, Beck Depression Inventory, Hamilton Rating Scale for Depression 17 item, Montgomery–Åsberg Depression Rating Scale, and Center for Epidemiologic Studies Depression Scale [22,25-28,32,33,36,40,42,45].

Furthermore, 2 trials, with apps for training in attention bias modification, claimed to be double blinded, with no further explanation on how blinding was ensured [43,44]. One trial blinded app allocation for patients when testing 2 different apps [45]. The remaining trials did not blind patients for the intervention (active vs control). In 12 trials, the authors explicitly stated that they had used blinded assessments for outcome measures [25-29,31-33,35,38,40,44]. Within these 12 trials, 5 trials used the patient-reported outcome as the primary outcome measure in nonblinded patients [25,28,29,32,33]. Although blinded assessors collected these data, answers to the questionnaires were self-reported. One trial tested for the success of blinding [32].

Information about recruitment, diagnosis, outcome measures, and how these were obtained can be found in Table 2.

Interventions

Although the included trials shared the concept of smartphones as a core feature of their interventions, the interventions were very different and tested in diverse settings with heterogeneous patient groups. Generally, 8 trials tested the app as a stand-alone treatment [35-37,44-48]. The remaining trials used variations of blended treatment and clinical support. Overall, 18 trials used prompts to engage users either from the app or by the investigators [22,26-28,30,32-44].

A glimpse of the diversity of interventions is presented here. Some trials used smartphones as a part of internet-based therapy, whereas others used smartphones as augmentation for face-to-face therapy or standard treatment programs. Most trials developed specific apps, whereas others used either commercially available or previously developed apps. Some interventions were interactive, whereas others had more static content. Some of the interventions made use of the unique possibilities a smartphone represents such as using global positioning system or allowing to interact with peers, relatives, or professionals, whereas others resemble classic internet therapies made for small smartphone screens to be more convenient and accessible to patients.

Control Group

One of the included trials involved the use of a placebo app, consisting of an inactive version of the software with limitations, but with no further description [41]. In addition, 2 trials (that both tested the effect of attention bias modification via a dot probe on the smartphone screen) used a different version of the app for the control group, setting the dot appearance at random instead of following a predefined pattern that was central to the treatment [43,44]. In 1 trial on bipolar disorder, the participants in the control group received a placebo smartphone without the app system [27]. No other trials mentioned attempts on placebo treatment. Overall, 5 trials used standard treatment as the comparator [24,27,35,38,40], and 11 trials used a waitlist control group [29,30,34,38,39,41,42,44-46,48]. Furthermore, 4 trials used some sorts of clinical intervention as the comparator [26,28,31,33], and 5 trials compared the intervention with another app [23,25,35,36,42].

Further description of the smartphone-based treatment interventions, availability of technology, and author affiliation with industry can be found in Table 3 and is summarized in Table 4.

Adherence to Smartphone Intervention

All but 2 trials [36,41] reported adherence by using one of the following or a combination: report on how many in-app lessons, modules or sessions participants completed, the number of daily or weekly users or logins, numbers of active users at a given time point, and composite scores measuring adherence.

Some trials included large tables on the usage of different components of the app, for example, the study by Schlosser et al [38], or included detailed program usage in the supplementary section, for example, the study by Stolz et al [39].

One trial economically compensated the participants for using the app [35]. Further comparison of adherence between trials was not feasible because of significant differences in how adherence was collected and reported in the individual trials.

Statistical Power Analysis

Overall, 11 trials based their required sample size on power analyses [24,27,28,30-33,39,40,45,48]. All but 1 [31] of these trials managed to recruit at least the desired number. In addition, 6 of the trials without power analyses were specified as pilot studies [22,35-37,42,46,47].

Statistical Analysis

All but 6 trials [22,36,37,41-43] used intent-to-treat principles in the primary analyses. Different methods were used to account for missing data. Some trials used multiple imputations [23,46,48], whereas others used mixed models [22,24,27,28,30,32,34,39,40,45] with missing at random or maximum likelihood estimations. One trial omitted data without imputation [41]. In addition, 8 trials included no explicit information on how the authors handled missing data [33,35-38,42,44,47]. Furthermore, 4 trials referred to an available predefined plan of analyses in the statistical section [27,32,33,35].

Technical Aspects of Smartphone-Based Treatment Interventions

One article informed about technical updates of the app, changes in interventions because of improvements, or technological problems that might have influenced the intervention over time [27]. None of the trials mentioned privacy breaches. In addition, 3 trials mentioned technical problems and how these affected interventions and the main hypothesis [23,40,48]. One trial collected automatically generated smartphone data (phone usage, social activity, and mobility) [40]. Furthermore, 8 trials collected data on app usage [24,33,34,42,43,45,47,48]. None of the trials accounted for changes or updates in the software or technical problems related to the intervention in the statistical analyses. Two trials included technology-specific baseline data or information about the participant’s technological abilities such as prior smartphone use or assessments of the participant’s ability to use a smartphone [32,33]. A total of 17 trials presented the intensity of use or user data with significant variations in usage among subjects and between trials [22-25,27,32-35,37-40,42,43,45,47].

Ethical Aspects

None of the included trials addressed the potential ethical aspects of using smartphones in the treatment of patients with a psychiatric disorder. Two trials included a section on ethics, including information on various trial registrations and approvals, data storage, and economic compensation [27,40].

Discussion

Principal Findings

This is the first systematic review regarding methodological challenges in RCTs investigating smartphone-based treatment interventions in patients with a psychiatric diagnosis. We included 27 trials with a wide range of psychiatric diagnoses and observed substantial between-trial heterogeneity. The trials were conducted in diverse settings and used different smartphone-based treatment interventions and different follow-up periods. The trials reported on various outcome measures, which, in nearly half of the trials, were not clearly predefined. Most trials only used unblinded patient-evaluated outcome measures. A single trial reported on the success of blinding procedures.

Furthermore, only 1 trial provided information regarding technological updates of the smartphone-based treatment intervention. A declaration of interests was missing in 9 of the trials. No trial compared participants with nonparticipants, thereby increasing the risk of selection bias and making generalization of the trial findings difficult.

The included trials used smartphones in various ways and applied treatments to very heterogenic populations. Generally, the combination of insecure diagnoses, lack of blinding, use of patient-evaluated outcome measures, and lack of trial protocols or thorough publicly available trial registrations implies that evidence on the effects and side effects is still warranted.

The included trials used very different comparators, and the treatments given, besides the intervention of interest, varied from nothing to intensive clinical standard treatment setups. Generally, important aspects concerning technological features and how these inevitably affect outcomes were sparingly reported. On the basis of the results of the review, in the following sections, we discuss the highlights and suggest recommendations for designing and conducting future RCTs investigating the effect of smartphones in psychiatry.

Inclusion Criteria of Trial Patients

Owing to the lack of diagnostic biomarkers within psychiatry, currently, the research-based clinical diagnostic process represents the golden standard. The use of online, patient-evaluated diagnoses will reduce the validity of the diagnoses and reduce the generalizability to the clinical practice of trial results.

The use of a research-based clinical diagnostic assessment such as the Schedules for Clinical Assessment in Neuropsychiatry providing Diagnostic and Statistical Manual of Mental Disorders and ICD-10 diagnosis [49] or another systematic diagnostic assessment system should be prioritized. If patients are thoroughly and validly diagnosed and characterized before inclusion in the RCT, that is, by their treating doctors or psychologists, the clinical diagnoses may as well be used, depending on the aims and hypotheses of the RCT.

Smartphones allow for a vast amount of data to be collected online or automatically. This could be used as an advantage when conducting large RCTs. Digital solutions let researchers reach more participants and make it easier for patients to participate in clinical trials, not having to show up in the research settings for diagnostic assessment, baseline data collection, and outcome assessments. This possibly facilitates broader and more feasible inclusion of patients. Questionnaires conducted on the screen of a smartphone might vary compared with validated paper-based questionnaires. However, a Cochrane review from 2015 concludes that “apps might not affect data equivalence as long as the intended clinical application of the survey questionnaire, its intended frequency of administration and the setting in that it was validated remain unchanged” [50]. Telephonic interviews are different from clinical evaluations conducted by clinicians with possibly more reduced validity of diagnoses and sociodemographic and clinical data. Nevertheless, telephonic interviews (by trained lay interviewers or professionals) have been used in the research of depression and anxiety disorders with reasonable validity of diagnosis [51].

Modern electronic data collection should be used wisely with an awareness of possible changes in the quality of the data that the researchers obtain.

Interventions and Comparators

This systematic review showed that a placebo-controlled design was used to a limited extent. There is no clear definition of what a digital placebo treatment can or should contain, and defining a proper placebo group in a nonpharmacological RCTs is always difficult. The expectation for technical solutions themselves to be helpful resembles a reaction that is comparable to receiving a placebo pill and has been suggested to affect patients independent of active treatment [52]. The term digital placebo has been suggested [52]. Still, little is known about this issue; however, ongoing RCTs are investigating this subject further using a sham app as a comparator [53].

Clear descriptions of the content of the interventions and comparators used by researchers in future RCTs should be prioritized and made available to readers. If not mentioned in the primary publication, clear reference to the description should be made, such as in the study protocol, earlier publication, appendices, or publicly available versions of the intervention used. Furthermore, researchers could beneficially design the active intervention to fit into clinical practice and adapt to the clinical settings either as a stand-alone treatment or in combination with clinical treatment and support.

Outcome Measures and Power Analyses

A clear predefined research question, represented by a predefined primary outcome measure in a precisely well-defined patient population, is necessary. Selecting a predefined and relevant primary outcome measure is crucial [12]. Assessors should be blinded to outcome measures. End points of clinical relevance, that is independent of or blinded to researchers, such as admittance to a hospital or relapse or recurrence of illness (as in an ongoing trial [54]), should be prioritized compared with biased end points [12]. Such outcomes benefit from being critical for patients, relatives, and clinicians.

Power analyses should preferably be made before start the trial to ensure that the required sample is realistic and able to answer the primary research question—often represented by significant changes in the primary outcome.

When using clinician-based outcome measures, this should preferably be done by blinded trained clinical researchers. This is a difficult task because patients may give the researcher a hint on their allocation. Precautions to hinder this should be taken and described.

Although patient-evaluated outcome measures are appealing because of the ease use, we should be careful when interpreting findings based on unblinded patient-evaluated outcomes as there will be a risk of bias.

Clinicians and administrators may play important roles in implementing such treatment systems following trial findings if the results indicate a possible effect of the new treatment [55].

Possible outcomes could include measures of clinicians’ and caregivers’ attitudes toward the intervention if involved. Further data regarding the use of resources and economic costs related to the intervention could be included if available.

Publication of a Trial Protocol and Reporting Guidelines

Compilation and publication of a trial protocol or a thorough description on publicly available registration sites, such as https://clinicaltrials.gov/, before analyzing data from the trial will increase the validity of the findings. Trial protocols and registrations should follow the standards for medical trial protocols [20] such as precise descriptions of inclusion and exclusion criteria, recruitment procedures, prioritized outcome measures including unblinded assessment, statistical power analyses, and a plan for analyses of participants versus nonparticipants. Furthermore, a thorough description of the used technology should be included [54,56,57]. Authors should publish deviations on especially the technical side of the protocol that were necessary, including valid arguments for the changes.

When reporting results, guidelines such as the CONSORT eHealth checklist [20] should be followed to increase transparency and for evaluation of findings.

Multidisciplinary Research

A close collaboration between information technology developers, clinicians and researchers is critical to ensure that the technology developed can answer the primary questions that are addressed and, vice versa, that the posed hypotheses match the available technology.

Authors should use the same transparency when working with the technology industry as researchers, doctors, and funders use with the medical industry. This systematic review shows that the authors’ declarations of conflicts of interest are frequently missing. The industry has a significant direct or indirect interest in developing app solutions and gathers information from patients as information in the digital economy involves major economic interests [58]. Any potential conflicts of interest should always be declared.

Technology in Randomized Controlled Trials

Smartphones represent a unique tool to measure adherence and fidelity in RCTs. However, there is no clear definition of how to measure or report adherence to smartphone-based treatment systems.

Smartphone solutions might assist in follow-up assessments and could advantageously use built-in features to ease the patients. Even when patients are “lost to follow-up,” assessments based on smartphone sensors could be collected, with patient permission, and might reflect behavioral changes for the user [59-61].

Prompts and reminders may comprise an important part of digital solutions to keep patients engaged in treatment.

These prompts, preferably, should be a part of the intended intervention that is investigated and not done only as part of the research project. The research team will potentially not be part of a real-world implementation of the technology and therefore give a false picture of adherence and effect of the intervention.

Information about technical skills and smartphone usage and habits might be valuable information and should be presented.

Reporting possible harms and adverse events are essential as the field of smartphone-based treatment is new, and little is known of potentially harmful effects such as worsening of symptoms or suicidal ideation. One trial has reported possible harms of using the intervention [27].

Potential harms and safety matters should be taken into consideration when designing and conducting the trial, as the example of the study by Stiles-Shields et al, [42] in which a safety protocol was used to standardize how researchers should react on suicidal ideations, expressed by participants either in the app or as part of the online questionnaires (where suicidal thoughts is a common item).

Updates, Revisions, and Adaptive Trials

Developing well-designed RCTs, including proper diagnostic procedures and thorough, robust, and blinded outcome measures, will inevitably be a time-consuming process. Still, it is crucial to test new smartphone-based treatment interventions on well-defined patient populations. Locking a digital intervention for several years is a challenge because technology would be expected to evolve and improve over time. In software engineering, updates and enhancements of the software are frequently released to fix bugs and errors and to improve usability, stability, robustness, and security. Moreover, the type of data collected from the phone via sensors and its processing by algorithms will also constantly change and adapt according to automatic machine learning models. Such learning systems will change based on new information and might be used increasingly in psychiatry as well as in many other fields [62]. Finally, the hardware and operating systems of the mobile phone regularly change, and the major vendors (Apple and Google) regularly release new hardware and operating systems that affect what an app can and cannot do.

Consequently, the concept of maintaining the digital intervention stable during a clinical trial is unrealistic, taking into consideration the long time span involved in conducting most RCTs.

To merge these opposing interests, we must accept these ongoing changes as a natural consequence of conducting research in the field, similar to surgeons who improve their skills or therapist who adapt and improve as they see more and more patients. Such updates should be reported systematically and should be thoroughly described so that it will be clear to readers which type of updates were made and what effect they might have had on trial results. In addition, the possible effects of add-ons could be presented and provided enough statistical power for subanalyses.

By allowing a more flexible approach to the RCT design, it would conform more to the so-called adaptive trial paradigm [17,63], and then RCTs might be able to answer questions regarding the effect of mHealth interventions without evaluating outdated technology [16,17].

It is necessary to accept and incorporate technological developments and changes in the design of a specific RCT in line with suggestions from similar but older scientific fields as telehealth in general [64].

Ethical Aspects

In a digital economy, data generated by apps and services constitute a valuable resource. Within psychiatry and other areas, such data can lead to discrimination of individuals and patient groups as a whole, and therefore, the handling and use of the data collected are of great importance. The value such information has to industry and other stakeholders is driving the many apps that are widely available without evidence of effects or possible harms. We need information obtained through well-designed and transparent trials to improve our knowledge of how these treatments can help patients and professionals.

Privacy, security, technology illiteracy, depersonalization of treatment systems, and technological paternalism are some of many possible ethical issues in this field [58].

These challenges apply both to the trial designs and the treatments being developed. The included trials hardly mentioned any such ethical issues.

Trials must consider and discuss possible ethical implications of the trial design and of the treatment itself.

Findings in a Scientific Context

The results from one of the included trials have been replicated using the same intervention, trial design, and outcome measures in a second trial only once [27,40]. However, this is necessary to increase evidence. Notably, numerous trial protocols within the field were identified during the search [54,65-69]. This indicates that the pace of the field is still increasing, and hopefully, future trials will provide more evidence within the field of smartphone-based treatment in patients with psychiatric disorders. This development is further shown in the large numbers of RCTs found in this review. Only 1 of the included trials in this review was published and included in a previous review from 2013 investigating the effects of smartphones in mental health [18]. With an increasing number of RCTs and hopefully increasing the quality of design and method, and a more uniform reporting of results, future meta-analyses of the effects of treatments will be possible.

Limitations

This study has various limitations: not all parts of the review process and data collection were double checked, no protocol of study method was published online beforehand, and no meta-analyses or statistical analyses were performed on the included data. We did not use any scoring systems for the assessments of quality and risk of bias in the included studies, leaving the assessment to the reader.

Conclusions

This first systematic review on the design, conduct, and methodological challenges of RCTs investigating the effect of smartphone-based treatment in patients with a psychiatric diagnosis suggests that there is a rapidly increasing interest for this type of treatment. Although an increasing number of trials tested new smartphone-based treatments, the trial designs and reporting were of low quality compared with more classic medical RCTs, and heterogeneity and methodological issues in individual trials limit the evidence.

Smartphone-based treatment interventions imply new challenges and opportunities, but we, as researchers, should consider strict methodological efforts when designing, conducting, and reporting such trials as in the rest of the field of medicine. Future trials employing strict methodology, including detailed description regarding patient recruitment, pre- and well-defined, prioritized outcome measures, information regarding technical updates and down periods, and statements on potential conflicts of interest are warranted. Research groups without trial experience should seek out information on how to conduct RCTs with high a methodological standard to ensure a high level of quality in the research. Finally, close collaborations between professions and specialties are needed in this complex branch of science.