This study uses PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines [25]. A systematic literature search was conducted using the electronic databases MEDLINE, Embase, Cochrane Library, PsycINFO, and CINAHL for relevant papers published from the inception of the databases through February 2020. Keywords related to sleep and mHealth were searched. To increase the coverage of relevant databases, we conducted a manual search of the Journal of Medical Internet Research and the Journal of Internet Interventions as well as the reference list of retrieved publications. See Multimedia Appendix 1 for an example of the search strategy.

The inclusion criteria consisted of peer-reviewed publications that (1) focused on mHealth apps that aimed to measure, track, or improve sleep; (2) described the design engineering, clinical implementation, or clinical evaluation of the apps; (3) focused on an app solely targeting sleep; and (4) targeted adults aged between 18 years and 60 years. The included studies were published in English.

Studies were excluded if they (1) focused on a sleep disorder other than insomnia; (2) focused on an app that provided multimodal interventions targeting health aspects other than sleep; (3) described internet, telephone, or text messaging interventions; or (4) were review papers. We excluded review papers that were at the early stages of screening; however, given the difficulty of discerning these papers, most were excluded in the full-text screening.

After duplicates were removed, a single author (MA) screened all the titles to identify potentially relevant studies. Abstract and title screen and full-text of potentially relevant studies were reviewed independently by 2 authors (MA and ES). Where there were conflicts, discrepancies were discussed, and consensus was reached with the senior author (NG).

A data extraction template was constructed by one of the authors (MA) to summarize the following characteristics of the studies: first author name, publication date, country, objective, study design, and sample. Full papers were then imported into NVivo (QSR International Pty Ltd, version 11.4.3, 2017) for detailed data extraction and thematic coding. A coding framework was developed by the authors for each of the following categories: design engineering and clinical implementation and evaluation. Relevant segments of text were coded and extracted into a Microsoft Excel spreadsheet. Risk of bias was not assessed, as this was deemed irrelevant to the aims of the paper.

Data were extracted and coded into the following subgroups:

The search strategy identified 6015 papers (Figure 1). Following the removal of duplicates (n=1503), 4512 papers were screened for title. Of those, 4246 were excluded, leaving 266 potentially relevant papers. Following a duplicate independent abstract review, 166 papers were excluded. The full text of 100 papers was obtained and independently reviewed by 2 authors (MA and ES). Following this, 82 papers were excluded, primarily for being review papers (n=24) or abstracts only (n=34), leaving 18 full-text papers to be included in this systematic review. A further 3 papers were excluded that reported evidence solely for concurrent validity (ie, comparison of app sleep tracking against objective measurements of sleep) and provided no further information about design engineering and clinical implementation and evaluation.

A summary of the study characteristics is presented in Table 1. There was a total of 15 papers. These papers assessed the design, implementation, and evaluation of 8 individual apps.

Furthermore, there were multiple studies (n=27) included within these 15 papers: 7 pre-post studies, 5 cross-sectional studies, 5 mixed methods studies, 5 concurrent validity studies, 2 randomized controlled trials (RCTs), and 3 descriptive studies (Table 1). Table 1 presents the papers and corresponding studies numbered under the column Number of studies and their objectives. Subjects were veterans or military service members (n=5) [30-34], healthy participants (n=6) [35-40], individuals with insomnia or sleep disturbances (n=4) [41-44], veterans affairs CBT-I clinicians (n=2) [45,46], and nurses with insomnia (n=1) [47]. The sample size varied among studies, ranging from 2 to 176. For longitudinal studies with over 10 participants, the mean study attrition rate was 25% (range 16%-39%).

Table 2 summarizes the 8 individual apps identified in this review: CBT-I Coach [33,34,45,46], Somnometer [35], Interactive Resilience Enhancing Sleep Tactics (iREST) [30,31], ShutEye [40], Sleepcare [44], Sleep Bunny [39], SleepFix [42,43], and 1 unnamed app [41].

In total, 6 of the 8 apps had a quantitative evaluation of treatment outcomes. Of the 11 studies that evaluated clinical effectiveness, the most frequently used sleep outcomes were self-reported sleep questionnaires, followed by app sleep diary measures and actigraphy (objective). In total, 8 of the 11 papers used the Insomnia Severity Index, the most widely used insomnia treatment outcome questionnaire to evaluate insomnia symptom severity [49] (Table 6). Only 2 papers focused solely on evaluating the effectiveness of the app [44,47].

This study provides the first comprehensive review of published studies and a framework for the design engineering and clinical implementation and evaluation of mobile apps for sleep disturbance. Despite the availability of over 500 [50] sleep mobile apps in commercial app stores (Apple App and Google Play Store), our review identified 15 papers assessing the design, implementation, and evaluation of 8 apps, only one of which was available to download on commercial app stores [32,33,40,45-47]. This means that less than 1% of all commercially available sleep apps have any published data on these aspects. Of the 15 papers, implementation metrics were reported in 10 papers and treatment outcomes were evaluated in 11 papers. Despite the potential of mHealth, there has been a small number of studies, a lack of standardization in design engineering approaches and clinical implementation assessments, and few comprehensive clinical evaluations.

For the increased utilization and adoption of mHealth apps, these technologies must be designed for people who will use them—both end users and multidisciplinary stakeholders [51-54]. Our review indicates that although some apps utilized best practice design approaches, for example, user-centered and multidisciplinary methods, approximately half of the apps did not report their design approach. We have previously shown how people with insomnia have unique user needs and preferences for sleep mobile apps that can drive engagement [42]; however, only 2 of the 8 apps reported any end-user involvement. Multidisciplinary teams are particularly crucial in a domain such as sleep, where various stakeholders (eg, clinical psychologists, sleep clinicians, psychiatrists) tend to be involved in patient care; however, only 2 apps reported this approach. Although these best practice design approaches may be an unspoken rule in app development, transparency in the reporting of these approaches is important in encouraging clinicians to be able to recommend such apps designed to sustain engagement.

There were few comprehensive evaluations of implementation, with only 3 out of 8 apps reporting more than 2 implementation metrics. Poor implementation can limit adoption or engagement, particularly in an uncontrolled and real-world setting, which in turn limits effectiveness. Our results suggest that although most apps had papers reporting some implementation outcomes, very few conducted a comprehensive exploration. For instance, although there is support for Sleepcare’s efficacy, there was a lack of a substantial assessment for its implementation. The National Health Service in the United Kingdom highlights the importance of implementation in the acknowledgment that their previous failure of digital health technology deployment was attributed to rushed and inadequate implementation [55]. Moreover, reporting of implementation processes may enable replication and reduce the gap between research and practice [56].

There was great heterogeneity among implementation studies in the conceptualization of implementation metrics. Given the inconsistency in terminology, we developed our own definitions to facilitate cross-study comparisons. This language incongruence highlights the need for a taxonomy of implementation metrics to clearly delineate key variables [57,58]. Similarly, there was a lack of standardized implementation metrics ranging from qualitative methods to nonstandardized quantitative surveys. This further contributes to blurring among constructs. When standardized measures were used, they were not specifically designed for mobile or sleep disturbances. For instance, the System Usability Scale was one of the most frequently used implementation scales in the identified studies [59]. Although it is a well-researched measure with good psychometrics [59], it is not designed for mobile or sleep use. A taxonomy of implementation outcomes with standardized tools for sleep mobile apps can advance the measurement and understanding of implementation processes.

Despite the importance of privacy and regulatory oversight in digital technology, only 3 papers mentioned privacy, and no papers considered mHealth regulations. Mobile devices collect a large amount of behavioral and health care data, raising concerns regarding mHealth app quality and safety [60-62]. Most mHealth apps do not have a transparent privacy policy, leaving end users unaware of what data are collected, how data are transferred, where data are stored, and with whom the data are shared [63]. Given that data security is a key concern among health care providers when recommending mHealth apps [64], transparent privacy policies and further regulatory oversight from bodies such as the FDA can combat this issue.

Of the evaluation studies, we identified 2 RCTs, only one of which was adequately powered according to the primary paper [44]. Most of the apps (6/8, 75%) delivered full or modified CBT-I (SRT or BTi), a well-established intervention where an evidence base already exists for face-to-face and internet-enabled delivery. CBT-I has been recommended as a first-line therapy for insomnia, given its substantial clinical base [8-10], and has shown comparable efficacy when delivered via the internet [65]. Although a previous systematic review demonstrated support for the efficacy of mobile phone interventions for sleep disorders and sleep quality, this review only included studies of 4 mobile apps [66]. In this review, only 2 of these 4 apps were included. One was excluded as it was a multimodal intervention and the other was targeted toward older adults. There is an evident need for methodologically robust and adequately powered studies assessing the effectiveness of mHealth apps for sleep disturbance. Nevertheless, mobile-delivered CBT-I has potential, given the therapy’s existing evidence base across various modalities.

Most apps described in this review were prototype versions, with only one app being matured to released (CBT-I Coach). This is aligned with a greater number of identified studies primarily focusing on earlier stages of development, for example, design or implementation, with some preliminary evaluations of efficacy. Of the apps identified in this review, only 1 app (Sleepcare) has the support of efficacy from an adequately powered RCT [44]. Despite this, there are no implementation studies for Sleepcare, and although acceptance is included as a measure in the paper, the results are not reported. Conversely, several papers described the design and implementation of the CBT-I Coach, but no full-scale efficacy evaluation was identified. However, CBT-I Coach is the only app available in commercial app stores.

Although it might be thought that a more mature app would have more cumulative evidence for its design, implementation, and evaluation, this does not prove to be the case. This mismatch in the maturity of sleep apps and the levels of available evidence ultimately reflects the unregulated nature of mHealth app development and deployment. To exacerbate the problem, the commercial app marketplace allows developers and researchers to freely release apps into these stores, which serve as the main app repository for consumers. Evidently, there is a need for a standardized set of evidence-based criteria for researchers to meet before making commercially available apps.

Ultimately, the lack of standardization in the evidence and reporting for the design engineering and clinical implementation and evaluation of mHealth apps for sleep disturbance stresses the need for a comprehensive framework to guide researchers and app developers. A recent systematic review highlighted the wide heterogeneity among the different published criteria for the assessment of mHealth apps, with 38 main classes of criteria [19]. Guidelines specific to the development and assessment of apps for sleep disturbance are particularly scarce. Establishing an extensive and standardized framework for mobile apps for sleep disturbance may lead to improved existing tools and the development of successful, high-quality, and effective apps.

As a first step, we developed a high-level framework based on the findings of this study to guide the design engineering and clinical implementation and evaluation of apps for sleep disturbance. The findings are summarized in Figure 2.

Although several frameworks exist for conditions such as PTSD [22], bipolar disorder [23], and hypertension [24], there are no frameworks for the development of sleep apps. Each of these frameworks similarly address design engineering, clinical implementation, and evaluation. For instance, the framework for bipolar disorder also notes the importance of designing with end users and multidisciplinary teams, addressing security and regulations with standards consistent with HIPAA, and evaluation with end users [23]. These frameworks were developed through a combination of lessons learned from firsthand app development, best practice principles, and theory-based design models. The framework in this study triangulates the findings from this review of digital sleep interventions to these previous frameworks augmented by our firsthand experience with the development of an app for insomnia [43,67]. Future work may consider adapting theory-based design models for sleep disturbance and integrating them into this framework.

Several reviews of commercial sleep apps have demonstrated a lack of validated sleep measurement algorithms [68], evidence-based principles for insomnia management [69], behavior change constructs [70], and overall low quality of functionality and content based on established app assessment criteria [71,72]. Evidently, commercial development of apps has severely outpaced academic research, putting their trustworthiness in question [73]. Our systematic search identified 13 clinical trial registrations, of which 6 were mobile apps not included in our systematic review as there were no available publications. Although partly attributable to the infancy of the mHealth field, there is still a necessity for timely and increased efforts of mobile sleep apps to progress to clinical evaluations. Collaboration between academia and industry may offer an opportunity to work together in developing scientifically rigorous solutions while keeping pace with the rapidly evolving app market.

This review has several limitations. First, we included English language publications only, which introduces publication bias, particularly given that these papers tended to originate from high-income countries such as the United States and Australia. Second, given that data extraction was based on the included studies only and that the mobile apps were not downloaded by the authors, some information such as app features and design approaches was not always clear or available. Third, given that our study focused on apps for sleep disturbance and did not include mHealth apps with multimodal interventions, including sleep, the inferences from this study may not extend to all sleep apps.

This is the first review to evaluate the design engineering and clinical implementation and evaluation of apps designed for sleep disturbance. It was found that despite a plethora of sleep apps available, there is limited research and a lack of standardization in the evidence base for the design, implementation, and evaluation of apps for sleep disturbance. Few apps had evidence for the use of best practice design approaches. Implementation assessments lacked standardization and consistency in implementation metrics used, and very few comprehensive efficacy evaluations were identified.

For the future development of engaging and evidence-based apps for sleep disturbance, we have developed a framework to guide the development and deployment process. The framework aims to address the need for (1) increased application and reporting of best practice design approaches, for example, user-centered and multidisciplinary teams; (2) comprehensive implementation assessments involving multiple metrics, tools validated for sleep, and privacy and regulatory considerations; and (3) rigorous evaluations of clinical efficacy. Collaboration between academia and the industry may facilitate the development of evidence-based apps in the fast-paced mHealth technology environment.