Published on 17.02.21 in Vol 23, No 2 (2021): February
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/24607, first published Sep 27, 2020.
Framework for the Design Engineering and Clinical Implementation and Evaluation of mHealth Apps for Sleep Disturbance: Systematic Review
Background: Mobile health (mHealth) apps offer a scalable option for treating sleep disturbances at a population level. However, there is a lack of clarity about the development and evaluation of evidence-based mHealth apps.
Objective: The aim of this systematic review was to provide evidence for the design engineering and clinical implementation and evaluation of mHealth apps for sleep disturbance.
Methods: A systematic search of studies published from the inception of databases through February 2020 was conducted using 5 databases (MEDLINE, Embase, Cochrane Library, PsycINFO, and CINAHL).
Results: A total of 6015 papers were identified using the search strategy. After screening, 15 papers were identified that examined the design engineering and clinical implementation and evaluation of 8 different mHealth apps for sleep disturbance. Most of these apps delivered cognitive behavioral therapy for insomnia (CBT-I, n=4) or modified CBT-I (n=2). Half of the apps (n=4) identified adopting user-centered design or multidisciplinary teams in their design approach. Only 3 papers described user and data privacy. End-user acceptability and engagement were the most frequently assessed implementation metrics. Only 1 app had available evidence assessing all 4 implementation metrics (ie, acceptability, engagement, usability, and adherence). Most apps were prototype versions (n=5), with few matured apps. A total of 6 apps had supporting papers that provided a quantitative evaluation of clinical outcomes, but only 1 app had a supporting, adequately powered randomized controlled trial.
Conclusions: This is the first systematic review to synthesize and examine evidence for the design engineering and clinical implementation and evaluation of mHealth apps for sleep disturbance. The minimal number of apps with published evidence for design engineering and clinical implementation and evaluation contrasts starkly with the number of commercial sleep apps available. Moreover, there appears to be no standardization and consistency in the use of best practice design approaches and implementation assessments, along with very few rigorous efficacy evaluations. To facilitate the development of successful and evidence-based apps for sleep disturbance, we developed a high-level framework to guide researchers and app developers in the end-to-end process of app development and evaluation.
J Med Internet Res 2021;23(2):e24607
Sleep disturbance is extremely prevalent and affects 33%-45% of adults . Insomnia is the most common sleep disorder, defined as a chronic and persistent difficulty falling asleep, maintaining sleep, or waking up too early [ ]. When left untreated, insomnia significantly increases the risk of adverse health outcomes, including mental health disorders [ , ], cardiovascular disease [ ], hypertension [ ], and diabetes [ ]. As insomnia poses serious risks to mental and physical health, exploring the efficacy of treatment is imperative. Cognitive behavioral therapy for insomnia (CBT-I) is an effective gold standard treatment that has consistently shown moderate to large treatment effects [ - ]. A limitation to the widespread use of CBT-I has been a shortage of adequately trained practitioners to treat the high volume of patients with insomnia.
One potential avenue for addressing these challenges is the use of digital therapy. The widespread adoption of mobile phones and apps can change the delivery of health care. This emerging field, known as mobile health (mHealth), refers to the provision of health care services and practice delivered using mobile technology. mHealth apps provide unique benefits in delivering health information and interventions, given the ubiquity, convenience, and affordability of mobile phones. Over 325,000 mHealth apps are available on app stores, and this number continues to grow rapidly . Research supports the utility of mHealth apps for a range of health issues, including depression, anxiety, schizophrenia, cardiac disease, physical activity, and diabetes [ - ]. The potential of mHealth is particularly significant for sleep disturbances as it presents a promising method for addressing this public health burden.
Health outcomes are not only dependent on the intervention delivered but also on the design engineering process employed. Design engineering combines design thinking, participatory design practices, software engineering methods, software, and quality assurance methods. Without analyzing this process, it is not possible to make inferences about the reasons for failure (eg, low engagement, lack of clinical efficacy) of an app. Alongside, this process involves the incorporation of clinically relevant content that either provides adjunctive or standalone therapy to traditional medical and psychological practice.
Clinical implementation requires consideration of end-user privacy and security, a major concern in mHealth. Recent work has demonstrated that mHealth apps routinely share and commercialize end-user data with third parties with very little transparency . In addition, there are several important regulatory considerations required to protect the public and ensure that mHealth apps meet the minimum requirements of quality. Regulatory bodies for mHealth include HIPAA (Health Insurance Portability and Accountability Act), which is a federal law mandating privacy and security standards, and the FDA (Food and Drug Administration), which evaluates the safety and marketing claims of mHealth apps. In the absence of oversight from these regulatory bodies, it is possible to misuse health care–related data [ ].
Despite the potential and ubiquity of mHealth apps, most apps lack evidence for their clinical efficacy among end users . Compounding this problem is a lack of framework to inform and standardize the process and reporting of design, development, and evaluation of mHealth apps [ ]. This may lead to clinical inefficacy, lack of medical condition–specific content, poor patient engagement, or even harmful apps [ , ].
Although various research groups have established frameworks for apps for posttraumatic stress disorder (PTSD) , bipolar disorder [ ], and hypertension [ ], there may be differences in the processes used according to the health condition, and there are no existing frameworks for sleep apps. This systematic review aims to assess the extent and nature of the peer-reviewed evidence and proposes a high-level framework for the design engineering and clinical implementation and evaluation of mHealth apps for sleep disturbance.
This study uses PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines . A systematic literature search was conducted using the electronic databases MEDLINE, Embase, Cochrane Library, PsycINFO, and CINAHL for relevant papers published from the inception of the databases through February 2020. Keywords related to sleep and mHealth were searched. To increase the coverage of relevant databases, we conducted a manual search of the Journal of Medical Internet Research and the Journal of Internet Interventions as well as the reference list of retrieved publications. See for an example of the search strategy.
The inclusion criteria consisted of peer-reviewed publications that (1) focused on mHealth apps that aimed to measure, track, or improve sleep; (2) described the design engineering, clinical implementation, or clinical evaluation of the apps; (3) focused on an app solely targeting sleep; and (4) targeted adults aged between 18 years and 60 years. The included studies were published in English.
Studies were excluded if they (1) focused on a sleep disorder other than insomnia; (2) focused on an app that provided multimodal interventions targeting health aspects other than sleep; (3) described internet, telephone, or text messaging interventions; or (4) were review papers. We excluded review papers that were at the early stages of screening; however, given the difficulty of discerning these papers, most were excluded in the full-text screening.
After duplicates were removed, a single author (MA) screened all the titles to identify potentially relevant studies. Abstract and title screen and full-text of potentially relevant studies were reviewed independently by 2 authors (MA and ES). Where there were conflicts, discrepancies were discussed, and consensus was reached with the senior author (NG).
Data Extraction and Coding
A data extraction template was constructed by one of the authors (MA) to summarize the following characteristics of the studies: first author name, publication date, country, objective, study design, and sample. Full papers were then imported into NVivo (QSR International Pty Ltd, version 11.4.3, 2017) for detailed data extraction and thematic coding. A coding framework was developed by the authors for each of the following categories: design engineering and clinical implementation and evaluation. Relevant segments of text were coded and extracted into a Microsoft Excel spreadsheet. Risk of bias was not assessed, as this was deemed irrelevant to the aims of the paper.
Data were extracted and coded into the following subgroups:
- Design engineering: This stage considers the therapeutic approach, design method, and features and functionalities of the apps. Although we appreciate the design engineering process as an ongoing and iterative process, for the purpose of this paper, we propose that this stage covers the point up until the prototype app is implemented among end users.
- Clinical implementation: This stage refers to the testing of the app among end users and includes an assessment of app maturity, implementation metrics (acceptability, engagement, usability, and adherence), and privacy (ie, data acquisition, use or disclosures of identifiable health data) and regulatory (eg, HIPAA) requirements. To categorize the implementation metrics used by the papers and uniformly compare evidence in the included studies, we developed definitions based on previous literature and adjusted for use with sleep apps [ - ]. Acceptability refers to how intended recipients react to the intervention, for example, interest, user satisfaction, and perceived appropriateness. Engagement refers to the usage metrics of the app. Usability refers to whether the app can be used as intended. Adherence refers to the degree to which the user follows the program and the prescribed recommendations of the therapy.
- Clinical evaluation: This stage refers to the assessment of treatment outcomes among end users.
The search strategy identified 6015 papers (). Following the removal of duplicates (n=1503), 4512 papers were screened for title. Of those, 4246 were excluded, leaving 266 potentially relevant papers. Following a duplicate independent abstract review, 166 papers were excluded. The full text of 100 papers was obtained and independently reviewed by 2 authors (MA and ES). Following this, 82 papers were excluded, primarily for being review papers (n=24) or abstracts only (n=34), leaving 18 full-text papers to be included in this systematic review. A further 3 papers were excluded that reported evidence solely for concurrent validity (ie, comparison of app sleep tracking against objective measurements of sleep) and provided no further information about design engineering and clinical implementation and evaluation.
A summary of the study characteristics is presented in. There was a total of 15 papers. These papers assessed the design, implementation, and evaluation of 8 individual apps.
Furthermore, there were multiple studies (n=27) included within these 15 papers: 7 pre-post studies, 5 cross-sectional studies, 5 mixed methods studies, 5 concurrent validity studies, 2 randomized controlled trials (RCTs), and 3 descriptive studies (). presents the papers and corresponding studies numbered under the column Number of studies and their objectives. Subjects were veterans or military service members (n=5) [ - ], healthy participants (n=6) [ - ], individuals with insomnia or sleep disturbances (n=4) [ - ], veterans affairs CBT-I clinicians (n=2) [ , ], and nurses with insomnia (n=1) [ ]. The sample size varied among studies, ranging from 2 to 176. For longitudinal studies with over 10 participants, the mean study attrition rate was 25% (range 16%-39%).
|Study||Country of origin||App name||Sample||Number of studies and their objectives||Study design|
|Aji et al (2019) ||Australia||SleepFix||Participants with insomnia or sleep disturbance|
|Aji et al (2020) ||Australia||SleepFix||Participants with insomnia|
|Babson et al (2015) ||United States||CBT-Ia Coach||Veterans with sleep disturbance and cannabis use disorder|
|Baron et al (2018) ||United States||Sleep Bunny||Healthy participants with sleep duration <7 hours|
|Bauer et al (2012) ||United States||ShutEye||Health participants|
|Horsch et al (2017) ||The Netherlands||Sleepcare||Participants with mild insomnia|
|Kang et al (2017) ||Korea||Unnamed||Participants with insomnia|
|Koffel et al (2016) ||United States||CBT-I Coach||Referrals from veterans affair medical center|
|Kuhn et al (2016) ||United States||CBT-I Coach||Veterans affairs CBT-I clinicians|
|Miller et al (2019) ||United States||CBT-I Coach||Veterans affairs CBT-I clinicians|
|Omeogu et al (2020) ||United States||CBT-I Coach||Female veterans administration nurses with insomnia|
|Pulantara et al (2018) ||United States||iRESTc||Active duty service members and veterans|
|Pulantara et al (2018) ||United States||iREST||Military service members and veterans|
|Reilly et al (2019) ||United States||CBT-I Coach||Veterans with insomnia|
|Shirazi et al (2013) ||Germany||Somnometer||University students|
aCBT-I: cognitive behavioral therapy for insomnia.
bRCT: randomized controlled trial.
ciREST: Interactive Resilience Enhancing Sleep Tactics.
summarizes the 8 individual apps identified in this review: CBT-I Coach [ , , , ], Somnometer [ ], Interactive Resilience Enhancing Sleep Tactics (iREST) [ , ], ShutEye [ ], Sleepcare [ ], Sleep Bunny [ ], SleepFix [ , ], and 1 unnamed app [ ].
|App name||Therapeutic approach||Design method||Level of automation|
|CBT-Ia Coach||CBT-I including sleep restriction therapy, stimulus control, psychoeducation, cognitive restructuring, and relapse prevention||•Multidisciplinary team, including experts in insomnia, VAb-trained CBT-I clinicians, clinical intervention development, technology, and implementation science|
• Clinician involvement in design
|Fully automated. However, it is not designed to replace clinician-delivered CBT-I; nonetheless, the app can be used as an educational resource|
|iRESTc||Traditional behavioral insomnia workflows, brief behavioral therapy for insomnia, a military study, and previous implementation of the JITAId platform||• Clinician involvement in design • Developed based on design principles: iterative and incremental development software model||High clinician input regarding treatment, information, recommendations, and messaging|
|ShutEye||Mobile, peripheral display to provide real-time sleep hygiene recommendations, relevant to the users’ set bed and wake time goals||N/Ae||Fully automated|
|Sleepcare||CBT-I including sleep restriction, relaxation, and sleep hygiene||• End-user involvement|
• Developed based on persuasive strategies: talk-and-tool design principle
|SleepFix||Sleep restriction therapy and stimulus control||• End-user involvement • Multidisciplinary team||Fully automated|
|Sleep Bunny||CBT-I and motivational interviewing and a telephone coaching manual developed from a web-based depression intervention||N/A||Fully automated|
|Somnometer||Social alarm clock app. Users rate their sleep quality and specify their sleep status, which they can share with their social network. Sleep duration is estimated based on the user’s phone interaction with app||N/A||Fully automated|
|Unnamed||CBT-I including sleep restriction therapy and stimulus control therapy||N/A||Clinician input required|
aCBT-I: cognitive behavioral therapy for insomnia.
bVA: veterans affairs.
ciREST: Interactive Resilience Enhancing Sleep Tactics.
dJITAI: just-in-time adaptive intervention.
eN/A: not applicable.
A total of 4 apps were identified to deliver a CBT-I intervention, 1 app used behavioral therapy for insomnia (BTi), 1 app delivered sleep restriction therapy (SRT), 1 app was a social alarm clock, and 1 app was a wallpaper display to promote healthy sleep behaviors. The average length of intervention (ie, use of app in the paper) was 5 weeks (mean 4.5. SD 1.5; median 4.5, range 3-6.5 weeks).
The apps were developed using various design approaches, including multidisciplinary (n=2), clinician involvement (n=2), user-centered (n=1), talk-and-tool design principle (n=1), and iterative and incremental development software model (n=1;). A total of 4 apps did not specify their design approach in their corresponding papers.
Features and Functionality
presents the features and functionality of the apps. All apps provided personalized sleep feedback (8/8, 100%). Most apps included psychoeducation or sleep hygiene (7/8, 88%), a sleep diary (6/8, 75%), reminders (5/8, 63%), and visualization features (5/8, 63%). In total, 5 of the 8 apps were fully automated and 3 required clinician input. Papers describing 6 apps provided end-user feedback regarding design [ , , , , , ]. End-user feedback was generally positive in nature, although some papers reported some negative feedback in the design process (4/6, 67%) [ , , , ].
|Feature||Apps in which the features were included, n (%)||Apps|
|Personalized sleep feedback||8 (100)||CBT-Ia Coach, iRESTb, ShutEye Sleepcare, SleepFix, Sleep Bunny, Somnometer, and unnamed app|
|Psychoeducation or sleep hygiene||7 (88)||CBT-I Coach, iREST, ShutEye, Sleepcare, SleepFix, Sleep Bunny, and unnamed app|
|Sleep diary||6 (75)||CBT-I Coach, iREST, Sleepcare, SleepFix, Sleep Bunny, and unnamed app|
|Reminders||5 (63)||CBT-I Coach, iREST, Sleepcare, Sleep Bunny, and SleepFix|
|Visualization||5 (63)||CBT-I Coach, iREST, SleepFix, Sleep Bunny, and Somnometer|
|Wearable synchronization||4 (50)||iREST, SleepFix, Sleep Bunny, and unnamed app|
|Relaxation||3 (38)||CBT-I Coach, Sleepcare, and unnamed app|
|Alarm||2 (25)||CBT-I Coach, and Somnometer|
|Clinician portal||2 (25)||iREST and unnamed app|
|Social features||2 (25)||SleepFix and Somnometer|
|Messaging with clinician||1 (13)||iREST|
aCBT-I: cognitive behavioral therapy for insomnia.
biREST: Interactive Resilience Enhancing Sleep Tactics.
All 8 apps included in this review had at least one paper assessing an implementation metric. A total of 10 out of the 15 papers reported on one or more of 4 implementation metrics: acceptability, usability, adherence, and engagement ().
|Attribute||Papers assessing attribute, n (%)||Described apps||Measures|
|Acceptability by end user||6 (40)||CBT-Ia Coach, ShutEye, Sleep Bunny, SleepFix, and unnamed app||Interview [, , , ], survey [ , , ], and MAUMb questionnaire [ ]|
|Acceptability by clinician||2 (13)||CBT-I Coach||17-item measure of clinicians’ retrospective perceptions of app  and 35-item measure of clinicians’ retrospective perceptions of app [ ]|
|Engagement||6 (40)||CBT-I Coach, iRESTc, SleepFix, Somnometer, and unnamed app||Self-reported number of sleep diaries [, ]; MAUM questionnaire measuring frequency of app use and length of use per session [ ]; number of sleep diary entries, mood logger entries, and game sessions measured by the app [ ]; number of scheduled alarms, posts shared on Facebook, and sessions measured by the app [ ]; and number of days the app was used, completion time for each log, number of devices with app accessing server each day measured by the app [ ]|
|Usability||4 (27)||CBT-I Coach, iREST, ShutEye, SleepFix, and Somnometer||System Usability Scale [, , ], modified Telerehabilitation Usability Questionnaire [ ], MAUM questionnaire [ ], qualitative survey [ ], and qualitative interview [ ]|
|Adherence||3 (20)||CBT-I Coach, ShutEye, Sleep Bunny, and Sleepcare||Average number of days and time spent on homework , patient adherence form (completed by clinicians) [ ], number of relaxation exercises performed [ ], deviation between real and agreed-upon time in bed [ ], number of coaching sessions completed [ ], and qualitative interview [ ]|
aCBT-I: cognitive behavioral therapy for insomnia.
bMAUM: Mobile App Use Measure.
ciREST: Interactive Resilience Enhancing Sleep Tactics.
Acceptability among end users and engagement were the most frequently assessed implementation metrics. Most papers measured acceptability through an interview or survey with end users and engagement using app-measured or self-reported app usage metrics. Usability was measured by less than half of the papers, with most using the System Usability Scale. Most apps had evidence available for 1 to 2 implementation metrics, and only 3 out of 8 apps (CBT-I Coach, ShutEye, and SleepFix) had papers reporting more than 2 implementation metrics.
The maturity levels of the apps ranged on a continuum of 3 levels: prototype, matured, and released . A prototype refers to a minimally viable product of the app with functionality that users can test. A matured version refers to an app that has undergone user testing and has been redesigned. A released version refers to an app that is available for download. Most apps were prototypes (n=5), and 1 app was a prototype-to-mature version ( ). Only 1 app was matured to release (CBT-I Coach). Furthermore, 1 study had no information regarding the maturity level of the app described. Only CBT-I Coach was available from the Google Play or Apple App Store at the time of writing the primary paper.
|Stage of maturity||App|
|Prototype||iRESTa, ShutEye, Sleep Bunny, SleepFix, and Somnometer|
|Prototype to matured||Sleepcare|
|Matured to released||CBT-Ic Coach|
|No information||Unnamed |
aiREST: Interactive Resilience Enhancing Sleep Tactics.
bNo apps met this maturity level.
cCBT-I: cognitive behavioral therapy for insomnia.
Only 3 apps considered the privacy of data collected [, , ]. In total, 3 out of the 4 apps that enabled wearable synchronization reported data relating to its use adjunct to the app [ , , ]. No studies have considered the regulatory requirements of the apps.
In total, 6 of the 8 apps had a quantitative evaluation of treatment outcomes. Of the 11 studies that evaluated clinical effectiveness, the most frequently used sleep outcomes were self-reported sleep questionnaires, followed by app sleep diary measures and actigraphy (objective). In total, 8 of the 11 papers used the Insomnia Severity Index, the most widely used insomnia treatment outcome questionnaire to evaluate insomnia symptom severity  ( ). Only 2 papers focused solely on evaluating the effectiveness of the app [ , ].
|Insomnia Severity Index||[- , , , , ]|
|Pittsburgh Sleep Quality Index||[, , , , , ]|
|Epworth Sleepiness Scale||[, , , ]|
|Dysfunctional Beliefs and Attitudes about Sleep-16||[, ]|
|Functional Outcomes of Sleep Questionnaire|||
|App sleep diary measures|
|Time in bed|||
|Total sleep time|||
|Sleep efficiency (percentage of time spent asleep while in bed)||[, , ]|
|Wake after sleep onset (wake time after initial sleep onset)|||
|Sleep onset latency (time taken to fall asleep)||[, ]|
|Sleep efficiency||[, ]|
|Total sleep time||[, ]|
|Stages of sleep|||
|Number of awakenings|||
|Wake after sleep onset|||
|Depression and anxiety|
|Hospital Anxiety and Depression Scale||[, ]|
|Center for Epidemiologic Studies Depression Scale|||
|PHQa-8, PHQ-9, or PHQ-15||[, ]|
|Generalized Anxiety Disorder Scale-7|||
|Beck Depression Inventory|||
|Beck Anxiety Inventory|||
|PTSD Checklist Civilian Version||[, ]|
|Pittsburgh Sleep Quality Scale (PSQI-Ac for PTSD)|||
|12-item Short Form Survey|||
|Flinders Fatigue Scale|||
|Clinical Global Impression–Improvement Scale|||
|West Haven-Yale Multidimensional Pain Inventory|||
|Pain Disability Index|||
|Marijuana smoking history|||
|Timeline follow-back interview|||
aPHQ: Patient Health Questionnaire.
bPTSD: posttraumatic stress disorder.
cPSQI-A: Pittsburgh Sleep Quality Index–Addendum.
This study provides the first comprehensive review of published studies and a framework for the design engineering and clinical implementation and evaluation of mobile apps for sleep disturbance. Despite the availability of over 500  sleep mobile apps in commercial app stores (Apple App and Google Play Store), our review identified 15 papers assessing the design, implementation, and evaluation of 8 apps, only one of which was available to download on commercial app stores [ , , , - ]. This means that less than 1% of all commercially available sleep apps have any published data on these aspects. Of the 15 papers, implementation metrics were reported in 10 papers and treatment outcomes were evaluated in 11 papers. Despite the potential of mHealth, there has been a small number of studies, a lack of standardization in design engineering approaches and clinical implementation assessments, and few comprehensive clinical evaluations.
For the increased utilization and adoption of mHealth apps, these technologies must be designed for people who will use them—both end users and multidisciplinary stakeholders [- ]. Our review indicates that although some apps utilized best practice design approaches, for example, user-centered and multidisciplinary methods, approximately half of the apps did not report their design approach. We have previously shown how people with insomnia have unique user needs and preferences for sleep mobile apps that can drive engagement [ ]; however, only 2 of the 8 apps reported any end-user involvement. Multidisciplinary teams are particularly crucial in a domain such as sleep, where various stakeholders (eg, clinical psychologists, sleep clinicians, psychiatrists) tend to be involved in patient care; however, only 2 apps reported this approach. Although these best practice design approaches may be an unspoken rule in app development, transparency in the reporting of these approaches is important in encouraging clinicians to be able to recommend such apps designed to sustain engagement.
There were few comprehensive evaluations of implementation, with only 3 out of 8 apps reporting more than 2 implementation metrics. Poor implementation can limit adoption or engagement, particularly in an uncontrolled and real-world setting, which in turn limits effectiveness. Our results suggest that although most apps had papers reporting some implementation outcomes, very few conducted a comprehensive exploration. For instance, although there is support for Sleepcare’s efficacy, there was a lack of a substantial assessment for its implementation. The National Health Service in the United Kingdom highlights the importance of implementation in the acknowledgment that their previous failure of digital health technology deployment was attributed to rushed and inadequate implementation . Moreover, reporting of implementation processes may enable replication and reduce the gap between research and practice [ ].
There was great heterogeneity among implementation studies in the conceptualization of implementation metrics. Given the inconsistency in terminology, we developed our own definitions to facilitate cross-study comparisons. This language incongruence highlights the need for a taxonomy of implementation metrics to clearly delineate key variables [, ]. Similarly, there was a lack of standardized implementation metrics ranging from qualitative methods to nonstandardized quantitative surveys. This further contributes to blurring among constructs. When standardized measures were used, they were not specifically designed for mobile or sleep disturbances. For instance, the System Usability Scale was one of the most frequently used implementation scales in the identified studies [ ]. Although it is a well-researched measure with good psychometrics [ ], it is not designed for mobile or sleep use. A taxonomy of implementation outcomes with standardized tools for sleep mobile apps can advance the measurement and understanding of implementation processes.
Of the evaluation studies, we identified 2 RCTs, only one of which was adequately powered according to the primary paper . Most of the apps (6/8, 75%) delivered full or modified CBT-I (SRT or BTi), a well-established intervention where an evidence base already exists for face-to-face and internet-enabled delivery. CBT-I has been recommended as a first-line therapy for insomnia, given its substantial clinical base [ - ], and has shown comparable efficacy when delivered via the internet [ ]. Although a previous systematic review demonstrated support for the efficacy of mobile phone interventions for sleep disorders and sleep quality, this review only included studies of 4 mobile apps [ ]. In this review, only 2 of these 4 apps were included. One was excluded as it was a multimodal intervention and the other was targeted toward older adults. There is an evident need for methodologically robust and adequately powered studies assessing the effectiveness of mHealth apps for sleep disturbance. Nevertheless, mobile-delivered CBT-I has potential, given the therapy’s existing evidence base across various modalities.
Most apps described in this review were prototype versions, with only one app being matured to released (CBT-I Coach). This is aligned with a greater number of identified studies primarily focusing on earlier stages of development, for example, design or implementation, with some preliminary evaluations of efficacy. Of the apps identified in this review, only 1 app (Sleepcare) has the support of efficacy from an adequately powered RCT . Despite this, there are no implementation studies for Sleepcare, and although acceptance is included as a measure in the paper, the results are not reported. Conversely, several papers described the design and implementation of the CBT-I Coach, but no full-scale efficacy evaluation was identified. However, CBT-I Coach is the only app available in commercial app stores.
Although it might be thought that a more mature app would have more cumulative evidence for its design, implementation, and evaluation, this does not prove to be the case. This mismatch in the maturity of sleep apps and the levels of available evidence ultimately reflects the unregulated nature of mHealth app development and deployment. To exacerbate the problem, the commercial app marketplace allows developers and researchers to freely release apps into these stores, which serve as the main app repository for consumers. Evidently, there is a need for a standardized set of evidence-based criteria for researchers to meet before making commercially available apps.
Ultimately, the lack of standardization in the evidence and reporting for the design engineering and clinical implementation and evaluation of mHealth apps for sleep disturbance stresses the need for a comprehensive framework to guide researchers and app developers. A recent systematic review highlighted the wide heterogeneity among the different published criteria for the assessment of mHealth apps, with 38 main classes of criteria . Guidelines specific to the development and assessment of apps for sleep disturbance are particularly scarce. Establishing an extensive and standardized framework for mobile apps for sleep disturbance may lead to improved existing tools and the development of successful, high-quality, and effective apps.
mHealth App Framework for Sleep Disturbance
As a first step, we developed a high-level framework based on the findings of this study to guide the design engineering and clinical implementation and evaluation of apps for sleep disturbance. The findings are summarized in.
Although several frameworks exist for conditions such as PTSD , bipolar disorder [ ], and hypertension [ ], there are no frameworks for the development of sleep apps. Each of these frameworks similarly address design engineering, clinical implementation, and evaluation. For instance, the framework for bipolar disorder also notes the importance of designing with end users and multidisciplinary teams, addressing security and regulations with standards consistent with HIPAA, and evaluation with end users [ ]. These frameworks were developed through a combination of lessons learned from firsthand app development, best practice principles, and theory-based design models. The framework in this study triangulates the findings from this review of digital sleep interventions to these previous frameworks augmented by our firsthand experience with the development of an app for insomnia [ , ]. Future work may consider adapting theory-based design models for sleep disturbance and integrating them into this framework.
Several reviews of commercial sleep apps have demonstrated a lack of validated sleep measurement algorithms , evidence-based principles for insomnia management [ ], behavior change constructs [ ], and overall low quality of functionality and content based on established app assessment criteria [ , ]. Evidently, commercial development of apps has severely outpaced academic research, putting their trustworthiness in question [ ]. Our systematic search identified 13 clinical trial registrations, of which 6 were mobile apps not included in our systematic review as there were no available publications. Although partly attributable to the infancy of the mHealth field, there is still a necessity for timely and increased efforts of mobile sleep apps to progress to clinical evaluations. Collaboration between academia and industry may offer an opportunity to work together in developing scientifically rigorous solutions while keeping pace with the rapidly evolving app market.
This review has several limitations. First, we included English language publications only, which introduces publication bias, particularly given that these papers tended to originate from high-income countries such as the United States and Australia. Second, given that data extraction was based on the included studies only and that the mobile apps were not downloaded by the authors, some information such as app features and design approaches was not always clear or available. Third, given that our study focused on apps for sleep disturbance and did not include mHealth apps with multimodal interventions, including sleep, the inferences from this study may not extend to all sleep apps.
This is the first review to evaluate the design engineering and clinical implementation and evaluation of apps designed for sleep disturbance. It was found that despite a plethora of sleep apps available, there is limited research and a lack of standardization in the evidence base for the design, implementation, and evaluation of apps for sleep disturbance. Few apps had evidence for the use of best practice design approaches. Implementation assessments lacked standardization and consistency in implementation metrics used, and very few comprehensive efficacy evaluations were identified.
For the future development of engaging and evidence-based apps for sleep disturbance, we have developed a framework to guide the development and deployment process. The framework aims to address the need for (1) increased application and reporting of best practice design approaches, for example, user-centered and multidisciplinary teams; (2) comprehensive implementation assessments involving multiple metrics, tools validated for sleep, and privacy and regulatory considerations; and (3) rigorous evaluations of clinical efficacy. Collaboration between academia and the industry may facilitate the development of evidence-based apps in the fast-paced mHealth technology environment.
Conflicts of Interest
MA, CG, RC, RG, and NG are named on 2 provisional patents for the SleepFix app.
Example search strategy.DOCX File , 12 KB
- Adams RJ, Appleton SL, Taylor AW, Gill TK, Lang C, McEvoy RD, et al. Sleep health of Australian adults in 2016: results of the 2016 Sleep Health Foundation national survey. Sleep Health 2017 Feb;3(1):35-42. [CrossRef] [Medline]
- Cuijpers P, Sijbrandij M, Koole SL, Andersson G, Beekman AT, Reynolds CF. The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisons. World Psychiatry 2013 Jun;12(2):137-148 [FREE Full text] [CrossRef] [Medline]
- Baglioni C, Battagliese G, Feige B, Spiegelhalder K, Nissen C, Voderholzer U, et al. Insomnia as a predictor of depression: a meta-analytic evaluation of longitudinal epidemiological studies. J Affect Disord 2011 Dec;135(1-3):10-19. [CrossRef] [Medline]
- Taylor DJ, Lichstein KL, Durrence HH, Reidel BW, Bush AJ. Epidemiology of insomnia, depression, and anxiety. Sleep 2005 Nov;28(11):1457-1464. [CrossRef] [Medline]
- Khan MS, Aouad R. The effects of insomnia and sleep loss on cardiovascular disease. Sleep Med Clin 2017 Jun;12(2):167-177. [CrossRef] [Medline]
- Vgontzas AN, Liao D, Bixler EO, Chrousos GP, Vela-Bueno A. Insomnia with objective short sleep duration is associated with a high risk for hypertension. Sleep 2009 Apr;32(4):491-497 [FREE Full text] [CrossRef] [Medline]
- Cappuccio FP, D'Elia L, Strazzullo P, Miller MA. Quantity and quality of sleep and incidence of type 2 diabetes: a systematic review and meta-analysis. Diabetes Care 2010 Feb;33(2):414-420 [FREE Full text] [CrossRef] [Medline]
- van Straten A, van der Zweerde T, Kleiboer A, Cuijpers P, Morin CM, Lancee J. Cognitive and behavioral therapies in the treatment of insomnia: a meta-analysis. Sleep Med Rev 2018 Apr;38:3-16. [CrossRef] [Medline]
- Jindal RD. Cognitive behavioral therapy alone and with medication for persistent insomnia. J Am Med Assoc 2009 Sep 09;302(10):1053. [CrossRef] [Medline]
- Mitchell MD, Gehrman P, Perlis M, Umscheid CA. Comparative effectiveness of cognitive behavioral therapy for insomnia: a systematic review. BMC Fam Pract 2012 May 25;13:40 [FREE Full text] [CrossRef] [Medline]
- 325,000 mobile health apps available in 2017. Research2Guidance. 2017. URL: https://research2guidance.com/325000-mobile-health-apps-available-in-2017 [accessed 2020-07-07]
- Wu Y, Yao X, Vespasiani G, Nicolucci A, Dong Y, Kwong J, et al. Mobile app-based interventions to support diabetes self-management: a systematic review of randomized controlled trials to identify functions associated with glycemic efficacy. JMIR Mhealth Uhealth 2017 Mar 14;5(3):35 [FREE Full text] [CrossRef] [Medline]
- Mateo GF, Font EG, Grau CF, Carreras XM. Mobile phone apps to promote weight loss and increase physical activity: a systematic review and meta-analysis. J Med Internet Res 2015 Nov 10;17(11):253 [FREE Full text] [CrossRef] [Medline]
- Creber RMM, Maurer MS, Reading M, Hiraldo G, Hickey KT, Iribarren S. Review and analysis of existing mobile phone apps to support heart failure symptom monitoring and self-care management using the mobile application rating scale (MARS). JMIR Mhealth Uhealth 2016 Jun 14;4(2):74 [FREE Full text] [CrossRef] [Medline]
- Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013 Nov 15;15(11):247 [FREE Full text] [CrossRef] [Medline]
- Firth J, Torous J. Smartphone apps for schizophrenia: a systematic review. JMIR Mhealth Uhealth 2015 Nov 06;3(4):102 [FREE Full text] [CrossRef] [Medline]
- Grundy Q, Chiu K, Held F, Continella A, Bero L, Holz R. Data sharing practices of medicines related apps and the mobile ecosystem: traffic, content, and network analysis. BMJ 2019 Mar 20;364:920 [FREE Full text] [CrossRef] [Medline]
- Armontrout J, Torous J, Fisher M, Drogin E, Gutheil T. Mobile mental health: navigating new rules and regulations for digital tools. Curr Psychiatry Rep 2016 Oct;18(10):91. [CrossRef] [Medline]
- Nouri R, R Niakan Kalhori S, Ghazisaeedi M, Marchand G, Yasini M. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc 2018 Aug 01;25(8):1089-1098 [FREE Full text] [CrossRef] [Medline]
- Buijink AWG, Visser BJ, Marshall L. Medical apps for smartphones: lack of evidence undermines quality and safety. Evid Based Med 2013 Jun;18(3):90-92. [CrossRef] [Medline]
- Chan S, Torous J, Hinton L, Yellowlees P. Towards a framework for evaluating mobile mental health apps. Telemed J E Health 2015 Dec;21(12):1038-1041. [CrossRef] [Medline]
- Schellong J, Lorenz P, Weidner K. Proposing a standardized, step-by-step model for creating post-traumatic stress disorder (PTSD) related mobile mental health apps in a framework based on technical and medical norms. Eur J Psychotraumatol 2019;10(1):1611090 [FREE Full text] [CrossRef] [Medline]
- Bauer AM, Hodsdon S, Bechtel JM, Fortney JC. Applying the principles for digital development: case study of a smartphone app to support collaborative care for rural patients with posttraumatic stress disorder or bipolar disorder. J Med Internet Res 2018 Jun 06;20(6):10048 [FREE Full text] [CrossRef] [Medline]
- Duan H, Wang Z, Ji Y, Ma L, Liu F, Chi M, et al. Using goal-directed design to create a mobile health app to improve patient compliance with hypertension self-management: development and deployment. JMIR Mhealth Uhealth 2020 Feb 25;8(2):14466 [FREE Full text] [CrossRef] [Medline]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):1000097 [FREE Full text] [CrossRef] [Medline]
- Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med 2009 May;36(5):452-457 [FREE Full text] [CrossRef] [Medline]
- Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011 Aug 05;13(3):52 [FREE Full text] [CrossRef] [Medline]
- WHO. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. WHO 2016:1-144 [FREE Full text] [CrossRef]
- De Geest S, Sabaté E. Adherence to long-term therapies: evidence for action. Eur J Cardiovascular Nursing 2016 Jun 22;2(4):323-323. [CrossRef]
- Pulantara IW, Parmanto B, Germain A. Clinical feasibility of a just-in-time adaptive intervention app (iREST) as a behavioral sleep treatment in a military population: feasibility comparative effectiveness study. J Med Internet Res 2018 Dec 07;20(12):10124 [FREE Full text] [CrossRef] [Medline]
- Pulantara IW, Parmanto B, Germain A. Development of a just-in-time adaptive mhealth intervention for insomnia: usability study. JMIR Hum Factors 2018 May 17;5(2):21 [FREE Full text] [CrossRef] [Medline]
- Reilly ED, Robinson SA, Petrakis BA, Kuhn E, Pigeon WR, Wiener RS, et al. Mobile app use for insomnia self-management: pilot findings on sleep outcomes in veterans. Interact J Med Res 2019 Jul 24;8(3):12408 [FREE Full text] [CrossRef] [Medline]
- Koffel E, Kuhn E, Petsoulis N, Erbes CR, Anders S, Hoffman JE, et al. A randomized controlled pilot study of CBT-I Coach: feasibility, acceptability, and potential impact of a mobile phone application for patients in cognitive behavioral therapy for insomnia. Health Informatics J 2018 Mar;24(1):3-13 [FREE Full text] [CrossRef] [Medline]
- Babson KA, Ramo DE, Baldini L, Vandrey R, Bonn-Miller MO. Mobile app-delivered cognitive behavioral therapy for insomnia: feasibility and initial efficacy among veterans with cannabis use disorders. JMIR Res Protoc 2015 Jul 17;4(3):87 [FREE Full text] [CrossRef] [Medline]
- Shirazi A, Clawson J, Hassanpour Y. Int J Human-Computer Stud. 2013. URL: https://doi.org/10.1016/j.ijhcs.2013.03.001 [accessed 2021-01-22]
- Bhat S, Ferraris A, Gupta D, Mozafarian M, DeBari VA, Gushway-Henry N, et al. Is there a clinical role for smartphone sleep apps? Comparison of sleep cycle detection by a smartphone application to polysomnography. J Clin Sleep Med 2015 Jul 15;11(7):709-715 [FREE Full text] [CrossRef] [Medline]
- Fino E, Plazzi G, Filardi M, Marzocchi M, Pizza F, Vandi S, et al. (Not so) Smart sleep tracking through the phone: findings from a polysomnography study testing the reliability of four sleep applications. J Sleep Res 2020 Feb;29(1):12935. [CrossRef] [Medline]
- Scott H, Lack L, Lovato N. A pilot study of a novel smartphone application for the estimation of sleep onset. J Sleep Res 2018 Feb;27(1):90-97 [FREE Full text] [CrossRef] [Medline]
- Baron KG, Duffecy J, Reid K, Begale M, Caccamo L. Technology-assisted behavioral intervention to extend sleep duration: development and design of the sleep bunny mobile app. JMIR Ment Health 2018 Jan 10;5(1):3 [FREE Full text] [CrossRef] [Medline]
- Bauer J, Consolvo S, Greenstein B. ShutEy: encouraging awareness of healthy sleep recommendations with a mobile, peripheral display. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery; 2012 Presented at: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; ; Austin, Texas, USA; May, 2012; Austin Texas USA p. 1401-1410. [CrossRef]
- Kang S, Kang JM, Cho S, Ko K, Lee YJ, Lee H, et al. Cognitive behavioral therapy using a mobile application synchronizable with wearable devices for insomnia treatment: a pilot study. J Clin Sleep Med 2017 Apr 15;13(4):633-640 [FREE Full text] [CrossRef] [Medline]
- Aji M, Gordon C, Peters D, Bartlett D, Calvo RA, Naqshbandi K, et al. Exploring user needs and preferences for mobile apps for sleep disturbance: mixed methods study. JMIR Ment Health 2019 May 24;6(5):13895 [FREE Full text] [CrossRef] [Medline]
- Aji M, Glozier N, Bartlett D, Peters D, Calvo RA, Zheng Y, et al. A feasibility study of a mobile app to treat insomnia. Transl Behav Med 2020 Mar 30:-. [CrossRef] [Medline]
- Horsch CH, Lancee J, Griffioen-Both F, Spruit S, Fitrianie S, Neerincx MA, et al. Mobile phone-delivered cognitive behavioral therapy for insomnia: a randomized waitlist controlled trial. J Med Internet Res 2017 Apr 11;19(4):70 [FREE Full text] [CrossRef] [Medline]
- Kuhn E, Weiss BJ, Taylor KL, Hoffman JE, Ramsey KM, Manber R, et al. CBT-I coach: a description and clinician perceptions of a mobile app for cognitive behavioral therapy for insomnia. J Clin Sleep Med 2016 Apr 15;12(4):597-606 [FREE Full text] [CrossRef] [Medline]
- Miller KE, Kuhn E, Owen JE, Taylor K, Yu JS, Weiss BJ, et al. Clinician perceptions related to the use of the CBT-I coach mobile app. Behav Sleep Med 2019;17(4):481-491. [CrossRef] [Medline]
- Omeogu C, Shofer F, Gehrman P, Green-McKenzie J. Efficacy of a mobile behavioral intervention for workers with insomnia. J Occup Environ Med 2020 Mar;62(3):246-250. [CrossRef] [Medline]
- Inal Y, Wake JD, Guribye F, Nordgreen T. Usability evaluations of mobile mental health technologies: systematic review. J Med Internet Res 2020 Jan 06;22(1):15337 [FREE Full text] [CrossRef] [Medline]
- Morin CM, Belleville G, Bélanger L, Ivers H. The Insomnia Severity Index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep 2011 May 01;34(5):601-608 [FREE Full text] [CrossRef] [Medline]
- Ko PT, Kientz JA, Choe EK, Kay M, Landis CA, Watson NF. Consumer sleep technologies: a review of the landscape. J Clin Sleep Med 2015 Dec 15;11(12):1455-1461 [FREE Full text] [CrossRef] [Medline]
- Maguire M. Methods to support human-centred design. Int J Human-Computer Stud. URL: https://doi.org/10.1006/ijhc.2001.0503 [accessed 2021-01-22]
- McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, et al. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol 2012;Suppl:49-56. [CrossRef] [Medline]
- Shah SGS, Robinson I, AlShawi S. Developing medical device technologies from users' perspectives: a theoretical framework for involving users in the development process. Int J Technol Assess Health Care 2009 Oct;25(4):514-521. [CrossRef] [Medline]
- Ben-Zeev D, Schueller SM, Begale M, Duffecy J, Kane JM, Mohr DC. Strategies for mHealth research: lessons from 3 mobile intervention studies. Adm Policy Ment Health 2015 Mar;42(2):157-167 [FREE Full text] [CrossRef] [Medline]
- NHS. Five year forward view. National Health Service. United Kingdom: NHS; 2014. URL: https://www.england.nhs.uk/wp-content/uploads/2014/10/5yfv-web.pdf
- Pfadenhauer LM, Mozygemba K, Gerhardus A, Hofmann B, Booth A, Lysdahl KB, et al. Context and implementation: a concept analysis towards conceptual maturity. Z Evid Fortbild Qual Gesundhwes 2015;109(2):103-114. [CrossRef] [Medline]
- Dubad M, Winsper C, Meyer C, Livanou M, Marwaha S. A systematic review of the psychometric properties, usability and clinical impacts of mobile mood-monitoring applications in young people. Psychol Med 2018 Jan;48(2):208-228. [CrossRef] [Medline]
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
- Brooke J. SUS: a 'Quick and Dirty' usability scale. In: Usability Evaluation In Industry. Florida: CRC Press; 1996.
- Powell AC, Landman AB, Bates DW. In search of a few good apps. J Am Med Assoc 2014 May 14;311(18):1851-1852. [CrossRef] [Medline]
- Kargl F, van der Heijden RW, Erb B, Bösch C. Privacy in mobile sensing. Digital Phenotyping and Mobile Sensing 2019:3-12. [CrossRef]
- Parker L, Bero L, Gillies D, Raven M, Grundy Q. The "Hot Potato" of mental health app regulation: a critical case study of the australian policy arena. Int J Health Policy Manag 2019 Mar 01;8(3):168-176 [FREE Full text] [CrossRef] [Medline]
- O'Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv 2019 Mar;15:110-115 [FREE Full text] [CrossRef] [Medline]
- Aguilera A, Muench F. There's an app for that: information technology applications for cognitive behavioral practitioners. Behav Ther (N Y N Y) 2012 Apr;35(4):65-73 [FREE Full text] [Medline]
- Zachariae R, Lyby MS, Ritterband LM, O'Toole MS. Efficacy of internet-delivered cognitive-behavioral therapy for insomnia - a systematic review and meta-analysis of randomized controlled trials. Sleep Med Rev 2016 Dec;30:1-10. [CrossRef] [Medline]
- Shin JC, Kim J, Grigsby-Toussaint D. Mobile phone interventions for sleep disorders and sleep quality: systematic review. JMIR Mhealth Uhealth 2017 Sep 07;5(9):131 [FREE Full text] [CrossRef] [Medline]
- Aji M, Gordon C, Peters D, Bartlett D, Calvo RA, Naqshbandi K, et al. Exploring user needs and preferences for mobile apps for sleep disturbance: mixed methods study. JMIR Ment Health 2019 May 24;6(5):13895 [FREE Full text] [CrossRef] [Medline]
- Ong AA, Gillespie MB. Overview of smartphone applications for sleep analysis. World J Otorhinolaryngol Head Neck Surg 2016 Mar;2(1):45-49 [FREE Full text] [CrossRef] [Medline]
- Yu JS, Kuhn E, Miller KE, Taylor K. Smartphone apps for insomnia: examining existing apps' usability and adherence to evidence-based principles for insomnia management. Transl Behav Med 2019 Jan 01;9(1):110-119. [CrossRef] [Medline]
- Grigsby-Toussaint DS, Shin JC, Reeves DM, Beattie A, Auguste E, Jean-Louis G. Sleep apps and behavioral constructs: a content analysis. Prev Med Rep 2017 Jun;6:126-129 [FREE Full text] [CrossRef] [Medline]
- Choi YK, Demiris G, Lin S, Iribarren SJ, Landis CA, Thompson HJ, et al. Smartphone applications to support sleep self-management: review and evaluation. J Clin Sleep Med 2018 Oct 15;14(10):1783-1790 [FREE Full text] [CrossRef] [Medline]
- Leigh S, Ouyang J, Mimnagh C. Effective? Engaging? Secure? Applying the ORCHA-24 framework to evaluate apps for chronic insomnia disorder. Evid Based Ment Health 2017 Nov;20(4):20. [CrossRef] [Medline]
- Wykes T, Schueller S. Why reviewing apps is not enough: transparency for trust (T4T) principles of responsible health app marketplaces. J Med Internet Res 2019 May 02;21(5):12390 [FREE Full text] [CrossRef] [Medline]
|BTi: behavioral therapy for insomnia|
|CBT-I: cognitive behavioral therapy for insomnia|
|FDA: Food and Drug Administration|
|HIPAA: Health Insurance Portability and Accountability Act|
|mHealth: mobile health|
|PTSD: posttraumatic stress disorder|
|SRT: sleep restriction therapy|
Edited by G Eysenbach; submitted 27.09.20; peer-reviewed by R Krukowski, Z Aghaei; comments to author 09.10.20; revised version received 12.10.20; accepted 15.01.21; published 17.02.21
©Melissa Aji, Christopher Gordon, Elizabeth Stratton, Rafael A Calvo, Delwyn Bartlett, Ronald Grunstein, Nick Glozier. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.02.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.