Published on in Vol 24 , No 3 (2022) :March

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27791, first published .
Implementation of Cognitive Behavioral Therapy in e–Mental Health Apps: Literature Review

Implementation of Cognitive Behavioral Therapy in e–Mental Health Apps: Literature Review

Implementation of Cognitive Behavioral Therapy in e–Mental Health Apps: Literature Review

Review

1Institute for Medical Informatics, Bern University of Applied Sciences, Biel, Switzerland

2Suchtfachklinik Zurich, Zurich, Switzerland

Corresponding Author:

Kerstin Denecke, Dr rer nat

Institute for Medical Informatics

Bern University of Applied Sciences

Quellgasse 21

Biel, 2501

Switzerland

Phone: 41 32 321 67 94

Email: kerstin.denecke@bfh.ch


Background: To address the matter of limited resources for treating individuals with mental disorders, e–mental health has gained interest in recent years. More specifically, mobile health (mHealth) apps have been suggested as electronic mental health interventions accompanied by cognitive behavioral therapy (CBT).

Objective: This study aims to identify the therapeutic aspects of CBT that have been implemented in existing mHealth apps and the technologies used. From these, we aim to derive research gaps that should be addressed in the future.

Methods: Three databases were screened for studies on mHealth apps in the context of mental disorders that implement techniques of CBT: PubMed, IEEE Xplore, and ACM Digital Library. The studies were independently selected by 2 reviewers, who then extracted data from the included studies. Data on CBT techniques and their technical implementation in mHealth apps were synthesized narratively.

Results: Of the 530 retrieved citations, 34 (6.4%) studies were included in this review. mHealth apps for CBT exploit two groups of technologies: technologies that implement CBT techniques for cognitive restructuring, behavioral activation, and problem solving (exposure is not yet realized in mHealth apps) and technologies that aim to increase user experience, adherence, and engagement. The synergy of these technologies enables patients to self-manage and self-monitor their mental state and access relevant information on their mental illness, which helps them cope with mental health problems and allows self-treatment.

Conclusions: There are CBT techniques that can be implemented in mHealth apps. Additional research is needed on the efficacy of the mHealth interventions and their side effects, including inequalities because of the digital divide, addictive internet behavior, lack of trust in mHealth, anonymity issues, risks and biases for user groups and social contexts, and ethical implications. Further research is also required to integrate and test psychological theories to improve the impact of mHealth and adherence to the e–mental health interventions.

J Med Internet Res 2022;24(3):e27791

doi:10.2196/27791

Keywords



Background

Mental disorders, including depressive and anxiety disorders, affect 29% of the global population in their lives [1]. Apart from the fact that mental disorders have an impact on people’s quality of life, they are one of the most common causes of occupational disability [2], resulting in high economic costs. The negative social aspects experienced by individuals with mental disorders include the inability to create and maintain lasting relationships and the stigmatization in society. These factors hamper individuals’ capacity to act and lead a self-determined life as members of society, discourage them from seeking professional help, and possibly reinforce the characteristics of mental disorders [3]. Mental disorders are usually treated using pharmacotherapy or psychotherapy [4]. However, there is a global shortage of mental health professionals as demand exceeds service provision. There are 9 psychiatrists per 100,000 people available in high-income countries [5], whereas there is 1 psychiatrist for every 10 million people in low-income countries [6]. In Europe, a comparative study between Finland and Spain—both with a similar prevalence of mental health disorders—showed a significant difference in the number of available staff resources. In Finland, for example, 13 psychologists were available per 100,000 inhabitants, whereas in Spain, only 2.9 psychologists were available per 100,000 inhabitants [7]. According to the World Health Organization (WHO), approximately 45% of people in high-income countries and 15% of people in low-income countries can access mental health services [8]. Leaving people with untreated mental disorders may increase suicide attempts and mortality [9]. Even if treated, approximately 98% of patients’ change processes induced by therapy occur outside of therapy sessions in their daily lives. Therefore, there is a need to provide support and self-help between therapy sessions, which increases the availability of cognitive behavioral therapy (CBT) to larger populations.

To address the issue of limited resources for treating individuals with mental disorders, e–mental health has gained interest in recent years, particularly for behavior change using elements of CBT and self-help (eg, MoodGym [10]). e–mental health is defined as “mental health services and information delivered or enhanced through the Internet and related technologies.” [11]. Numerous studies have shown that e–mental health interventions are comparable in effectiveness to traditional face-to-face psychotherapy [12,13], thus providing a possible solution for people who do not have access to face-to-face therapy. e–Mental health enables users to learn more about their mental health condition through self-help services; it empowers them to strengthen their self-management and improve their health, sometimes including peer-to-peer support [14]. e–Mental health, realized as mobile health (mHealth) apps, aims to expand the availability and quality of mental health treatment. mHealth apps often ask users to enter data for reflection and awareness and provide relevant information depending on user input. Sometimes, they also collect data from wearables. The number of apps addressing mental health has rapidly increased in recent years [15,16]. Details on the technical implementation of mHealth apps are rarely described in scientific papers [17], although implementation is of utmost importance to enable patient agency and facilitate self-therapy practices. This study aims to investigate which CBT techniques are implemented by which technologies in mHealth apps and derive the research gaps.

CBT and mHealth Apps

CBT is an “active, problem-focused, and time-sensitive approach to treatment that aims to reduce emotional distress and increase adaptive behavior in patients with a host of mental health and adjustment problems” [15,18]. There are four fundamental techniques of psychotherapy used in CBT: cognitive restructuring, behavioral activation, exposure, and problem solving [15,18]. In cognitive restructuring, therapists support patients in recognizing, evaluating, and modifying maladaptive or unhelpful thinking. Behavioral activation helps patients to actively re-engage in their lives. Exposure comprises systematic contact with a feared stimulus, whereas problem solving aims to help patients identify and implement solutions to their problems. We based our work on the implementation of CBT in mHealth apps on these 4 fundamental techniques of psychotherapy.

The efficacy of CBT has been demonstrated in multiple forms of psychopathology, including anxiety disorders, depression, and eating disorders [19]. Efficacy in the context of this study means that the effectiveness of an intervention can be demonstrated. mHealth apps provide options for practices, which were formerly elements of therapist–patient interaction; thus, this provides momentum for new routines and social forms [20] of coping with mental health problems.

In this study, we aim to assess which technologies are used in mHealth apps to implement the CBT technique. More specifically, we seek to answer the following research questions: for which mental illnesses have CBT-based apps been in use or tested, which CBT techniques are implemented in mHealth apps, which technologies are used to implement CBT techniques in mHealth apps, and which research gaps exist in mHealth apps for realizing CBT?


Overview

We answered our research questions using a literature search and review. We studied which of the 4 fundamental CBT techniques (cognitive restructuring, problem solving, behavioral activation, and exposure) have been implemented in mHealth apps. Furthermore, we investigated the technologies used for their implementation.

Search Strategy

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) criteria guided the conduct and reporting of our literature search [21]; for the PRISMA checklist, refer to Multimedia Appendix 1. The search was conducted between June 6 and June 13, 2020, considering all articles published during the period of 2007 to 2020, as the first iPhone was launched in 2007, establishing the technological basis for mobile apps. A total of 3 databases were consulted to find the relevant papers. Papers included in PubMed were retrieved using the search string described in Textbox 1. As we were also interested in the technical aspects of CBT in mHealth apps, we additionally searched the libraries of IEEE Xplore and ACM Digital Library (Textbox 1).

Search strings used for database search.

Search strings used to search PubMed

  • Cognitive behavioral therapy AND mental health AND (telemedicine ORmobile health OR mhealth OR smartphone) NOT (internet delivered OR internet-delivered): 287 results with abstract

Search strings used to search IEEE Xplore

  • Cognitive behavioral therapy: 65 results

Search strings used to search ACM Digital Library

  • Query: (Abstract: cognitive behavioral therapy AND mental health) AND (Abstract: mobile health OR mhealth OR smartphone); Filter: (Article type: Research Article, Publication date: [1/1/2007 TO *], ACM content: DL): 178 results
Textbox 1. Search strings used for database search.

Inclusion and Exclusion Criteria

Articles were included in this review if they were dealing with CBT and mHealth apps, they were primary studies reporting results, and adults (aged >18 years) were the target population.

Articles were excluded if the target populations for interventions were military veterans, children, or adolescents. These target groups differ from the general public. Military veterans have a significantly higher prevalence of posttraumatic stress disorder than the average population [22] and often receive care in specialized institutions. On the other hand, digital resources for children or adolescents need to address specific cognitive and developmental issues [23] and cannot be directly compared with apps for adults. Therefore, we decided to exclude apps that explicitly targeted these groups from this review. Furthermore, all papers dealing only with web-based interventions and those describing only the study protocol without the final results were excluded.

Eligibility and Data Extraction

To assess the eligibility of the articles, all titles and abstracts were examined by 2 independent reviewers (KD and SN) in the first round. In the second round, the full texts of the selected articles were extracted and carefully analyzed to confirm their eligibility. Eligibility doubts were discussed until an agreement was reached. The selected articles were included in the qualitative synthesis.

Two reviewers (KD, NS) extracted data from the selected studies regarding CBT techniques, technologies implemented in the mHealth app, type of mHealth app, considered medical conditions, and outcome. With CBT techniques, we referred to the 4 fundamental techniques of psychotherapy applied within CBT (cognitive restructuring, behavioral activation, exposure, and problem solving [15]). With technologies in mHealth apps, we implied technical means used in the apps for realizing specific functionalities. More specifically, we assessed the provision of audio and video content, interactive elements (eg, communication facilities with humans or computer systems such as chatbots), social network technologies, gamification, and automatic analysis facilities (eg, for sentiment or emotion detection, recommendation, and text analysis). Data were abstracted into a spreadsheet standardized for this review. Finally, we derived the research gaps in e–mental health from the results.


Sample

A total of 530 papers were retrieved by our search, as follows: 287 (54.2%) records in PubMed, 65 (12.3%) in IEEE Xplore, 178 (33.6%) in ACM Digital Library, and no duplicates. Of these 530 papers, 34 (6.4%) papers met the inclusion criteria and were, therefore, included in the qualitative synthesis (see the flowchart of the selection procedure in Figure 1). Papers were excluded during eligibility screening and data extraction if they described only mock-ups or no apps.

Figure 1. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart of the selection procedure.
View this figure

Characteristics of Studies

Most papers (21/34, 62%) reported randomized controlled trials (RCTs) in which ≥1 mental health app was tested (Multimedia Appendix 2). The remaining papers used different study designs. Of the 34 papers, 4 (12%) only assessed usability through surveys. For RCTs, the number of involved participants varied between 30 and 1098, with an average of 333. The studies designed as surveys involved an average of 135 participants. We found the largest number of participants (n=3977) in an observational study [24]. This study did not recruit participants explicitly but analyzed the use protocols of the app under consideration. It remains unclear whether these protocols originated from people with diagnosed mental health problems.

Most studies considered usability [25,26], user satisfaction, acceptance [24], adherence [27], or engagement as outcome measures. RCTs mainly studied efficacy, adherence, or acceptability compared with a control group. In RCTs, e–mental health interventions were often compared against each other instead of comparing the app intervention with standard face-to-face psychotherapy. For example, the efficacy of a web-based implementation of the Kokoro app was compared with that of an app-based intervention [28]. Among other things, outcome measures included the Patient Health Questionnaire–9 or Emotional Self-awareness Scale [29]. Of the 34 papers, 3 (9%) studied the Kokoro app for depression. In these papers, the effectiveness of the app-based intervention was reported; the use of the app reduced the symptoms of depression [28,30,31].

Mental Health Conditions Covered

Most studies (32/34, 94%) focused their research on a user group with a specific disorder. We classified the disorders according to the WHO International Classification of Diseases–10th revision (version 2019) coding system (Table 1). Most studies (16/34, 47%) used apps that offered support for persons with depression. Furthermore, there was a wide range of apps that targeted anxiety disorders; addiction problems; and psychiatric disorders such as schizophrenia, bipolar disorders, or suicidal tendencies. These disorders are often treated by CBT, which can be realized in an mHealth app in contrast to other types of therapies.

Table 1. Health conditions examined in the studies (N=34).
Health conditionWHOa ICD-10b codeTotal, n (%)
Drug addiction—cannabisF12.11 (3)
AgoraphobiaF40.01 (3)
Borderline personalityF60.31 (3)
DepressionF3216 (47)
Eating disorderF502 (6)
InsomniaG471 (3)
PsychosisF292 (6)
Schizophrenia, schizoaffective, or bipolar 1F20, F25, and F311 (3)
SmokingZ72.03 (9)
StressF993 (9)
SuicidalityR45.81 (3)
UnknownN/Ac2 (6)

aWHO: World Health Organization.

bICD-10: International Classification of Diseases–10th revision.

cN/A: not applicable.

CBT Techniques in mHealth Apps

Apps may have integrated techniques to address different purposes. We classified the main purposes of the apps as informational, coach, or therapy. An app was considered informational when it mainly provided information to the user, for example, on mental health. Apps were classified as coaches when suggestions were tailored to the user and their specific mental health condition or when support in managing a mental illness was delivered. As therapeutic apps, we grouped apps that delivered CBT with the aim of creating a therapeutic setting.

We could not identify any apps that were only informational. Most apps provided information but had additional functionalities that led to their classification as coach or therapy. Most (20/34, 59%) apps could be considered as coaches in mental health. Of the 34 apps, 13 (38%) were classified as therapy; for 1 (3%) app, this classification was not applicable. We concluded that CBT-based mHealth apps go beyond patient education. They support and extend their self-help capabilities.

We identified in the included papers 3 of the 4 fundamental techniques of psychotherapy applied in CBT (cognitive restructuring, behavioral activation, exposure, and problem solving [15]; Multimedia Appendix 3 [24-27,29,30,32-50]). mHealth apps provided toolkits for cognitive restructuring, which included diary-keeping functionalities and support in changing thoughts or tensions. Behavioral activation was realized by providing information on mental health conditions by tracking activities or setting goals. Assigning homework to patients, which is the best practice as a problem-solving technique [15], was realized in many apps (20/34, 59%) by providing exercises and activities as strategies to cope with mental health problems. Direct communication with a therapist, which is not a CBT technique itself but helps in standard setting for delivering CBT, was enabled by 6% (2/34) of the apps included in the review.

Technologies in mHealth Apps for Implementing CBT

When developing mHealth apps for CBT, it is important to know which technologies are useful and efficient for implementing different CBT techniques. Interactive elements were included in 35% (12/34) of apps to realize behavioral activation and cognitive restructuring. Among other things, we could identify the following interactive elements: automatic question answering functionalities, message exchange with the treatment team, a virtual character that provided information or explanations, personalization, and persuasion methods (Multimedia Appendix 4 [24-28,31-37,40-46,50-54]). Of the 34 apps, 3 (8%) provided a conversational agent or chatbot with which the user could communicate. Chatbot technology was used to support cognitive restructuring. Methods for automatic analysis of free textual input (natural language processing) and machine learning methods (eg, text classification or clustering) were used to understand user input or personalize recommendations according to user input. However, these methods are still rarely used (3/34, 9% papers). Audio and video were used to provide information or demonstrate exercises such as meditation exercises (ie, support behavioral activation and problem solving). Few apps calculated scores such as sentiment or emotion scores that were shown on a timeline. Sentiment or emotion scores quantify the sentiment or emotion of a user (eg, positive and negative sentiment or the strength of an emotion such as sadness). The following two technologies were integrated into some apps that aimed to improve the user experience and adherence: gamification and social networks. Social aspects were integrated, enabling the user to connect with other users. This social community aspect was interesting in the context of depression but also in other mental diseases, where people often experience loneliness and a lack of social contact. Gamification and social networks integrated into these apps did not aim to deliver CBT but, more importantly, aimed to increase the adherence and attractiveness of using the apps [32,51]. Gamification such as collecting jigsaws was used in 24% (8/34) of apps, and audio or video content was provided by 21% (7/34) of apps. Approximately 15% (5/34) of apps enabled connections with social networks and communities.


Principal Findings

Overview

The main finding of this review is that mHealth apps for CBT exploit two groups of technologies: (1) technologies that implement concrete CBT techniques for cognitive restructuring, behavioral activation, and problem solving and (2) technologies that aim to increase user experience, adherence, and engagement. The latter tries to address the current challenge in delivering CBT, which is insufficient adherence to CBT techniques [55]. A CBT technique that we did not find implemented was exposure. There is a broad range of technologies used in mHealth apps to deliver CBT; however, no technology was used in all apps. However, a trend toward the use of interactive elements, gamification technologies, and technologies supporting social activities could be recognized. No app was purely informational; however, most could be classified as coaches and a few as therapeutic. There have been attempts to deliver therapy through such apps [33-35,52]. Most apps target patients with disorders that are often treated with CBT, such as depression; anxiety disorders; addiction problems; and psychiatric disorders such as schizophrenia, bipolar disorders, or suicidal tendencies.

Mental Illnesses for Which CBT-Based Apps Have Been Used or Tested

Depression was reported most often in the assessed studies. Depressive disorders are widespread in all countries and comorbid with other mental diseases. According to the WHO, >264 million people of all ages experience depression worldwide. Thus, the target user group is huge. Depression is a leading cause of disability worldwide and a major contributor to the overall global burden of disease. There seems to be a burden of being able to deliver support to those people, and mental health apps are obviously comprehensively tested to fill this gap. Although our review did not deliver much evidence on the efficacy of mHealth apps in the mental health context, Khademian et al [56] found that mHealth apps that provide behavior change strategies, such as CBT and techniques for behavioral activation, have significant effects on depression, anxiety, and stress.

Psychiatric diseases such as psychosis or schizophrenia were also targeted. A challenge with psychiatric diseases is that critical situations can occur where professional reactions are essential (eg, to prevent a suicide attempt [57]). mHealth apps that deliver predefined content cannot react individually to various situations or specific user needs. There is a need for mHealth apps without undesired side effects; that is, those able to respond appropriately in situations of crisis [17].

CBT Techniques Implemented in mHealth Apps

We found that some methods used in standard CBT were implemented in mHealth apps. Sometimes, it was simply a digitization of the existing technique; however, there were cases where mHealth apps offered some benefits compared with the traditional therapy setting. Thought records or coping cards are methods used within standard CBT to achieve cognitive restructuring [15]. These methods are often provided in mHealth apps, for example, by supporting the keeping of diaries.

Scheduling and monitoring activities are the central components of behavioral activation in CBT [58]. Behavioral activation recommends planning activities in the evening before or in the early morning. As therapy sessions are spread across weeks and planning tasks are conducted with the therapist, day-to-day planning is not practiced in traditional therapy settings. With the use of mHealth technologies, daily planning and engagement can be supported, and personalized recommendations can be made [36]. Furthermore, increasing patients’ confidence in their ability to cope with stress and adversity and their overall coping skills is a major factor in mHealth apps contributing to the effects on mental health [29].

Information provision as a means of behavioral activation is another important aspect realized in many apps. This task includes teaching patients the basics of their mental health condition and CBT techniques. Morriss et al [59,60] have shown that knowledge and understanding of the medical condition are effective in supporting the everyday coping of patients with mental illness and can foster compliance of patients to the health intervention. Furthermore, patients become more competent in making decisions related to their health through information provision [61]. As providing information is a repetitive task for health care providers, an app providing information and teaching patients on the basics of mental health could save time for health care providers. Furthermore, the patient can read the information several times. However, studies show that e–mental health apps often contain incorrect information [62] and are not necessarily evidence based.

Assigning homework to patients is the best practice as a problem solving technique [15]. This aspect of CBT, which is available in many apps, is realized by providing exercises and activities. Exposure (imaginal exposure and interoceptive exposure [15]) is a CBT technique that was not represented at all in mental health apps included in our review. However, recent research has exploited virtual reality for realizing exposure apps; for example, to reduce the fear of heights [63]. It is obvious that there is still potential for developing and testing mHealth apps that target exposure techniques.

In conclusion, mental health apps that provide CBT support the performing of repetitive tasks such as keeping a diary or exercising, which are essential tasks and techniques within CBT for cognitive restructuring, behavioral activation, and problem solving [15]. Although these tasks must be realized by patients as part of their standard therapy, even without mHealth support, their digitization in mHealth apps can increase adherence, user engagement, and retention or facilitate learning [64]. A mobile app is available 24 hours a day. Thus, monitoring and tracking can be performed at any time and at any place. Records in the app cannot get lost as paper-based records do. Reminders can be sent to the user on a regular basis to ensure that records are made or activities come into the user’s mind [37].

Technologies Used to Implement CBT in mHealth Apps

Most apps often present information using videos or audio, or they collect data on activities from a user for behavioral activation. Social aspects are integrated, enabling the user to connect with other users. This social community aspect is interesting not only in the context of depression but also other mental diseases, where people often experience loneliness and a lack of social contact. Gamification and social networks integrated in these apps do not aim to deliver CBT but, more importantly, aim to increase the adherence and attractiveness of using these apps [37]. Artificial intelligence, including natural language processing, language understanding, and chatbot technology, has been rarely used in available apps. A reason might be that unforeseen errors or reactions can occur when the system misinterprets the user input, which might be avoided [65]. Other studies have shown an increased interest in mental health chatbots [66] or that chatbots are examples of the next generation of mental health [17]. However, this was not reflected in our review. Another reason for the limited inclusion of artificial intelligence in mental health apps may be related to medical device regulation, which requires traceability and increases the demand on the development process.

Research Gaps That Exist in mHealth Apps for Realizing CBT

After scoping the landscape of mHealth apps that implement CBT, we summarized the open research issues. There is a need for the following: RCTs studying the efficacy of the single technologies implemented in mHealth apps for realizing CBT techniques; studies on the impact on patients’ agency, including trust and overreliance; consideration of psychological theories during mHealth implementation to increase impact and adherence; and support in recommending or selecting CBT apps as health interventions.

Although more than half of the analyzed studies reported on RCTs, which is the state-of-the-art study design for proving the efficacy of medical interventions [67], these trials often did not assess the efficacy of the mHealth intervention. Similar results have been reported by Bauer et al [68]. For chatbots in clinical psychology, Bendig et al [17] noted that studies on mental health chatbots mainly assessed feasibility and acceptance. Studies are needed to assess which technology is well-suited for implementing a particular CBT technique, as well as for which mental disorders are mHealth apps efficient. This will help derive the best practices for implementing CBT techniques in mHealth. In this context, it is also important to clarify the role of mHealth apps; for example, whether they are intended to accompany the therapy or fill the treatment gap until a therapy can be started.

We recognize that the impact of mHealth apps on patients’ agency is not explicitly considered in these studies. Agency is generally viewed as the “capacity to act, produce and anticipate a desired outcome within a particular context” [69]. In our context, this means, among other things, that through mental health apps, patients would begin to manage themselves as participants with agency exercised on the basis of patient autonomy [70].

From a technical perspective, we want to achieve good adherence and frequent use of apps and support this by integrating features such as gamification to encourage frequent use. However, there are also concerns about overreliance on mental health apps, particularly for people who already have trouble with addictive web-based behavior [71]. The use of smartphone apps might be problematic, given specific mental conditions such as anxiety or depression [72]. Studies that assess this risk of overreliance in long-term use of mHealth apps, as well as assessments related to trust in mHealth from perspectives of both patients and therapists, are missing. There are indications that users think they have even more trust in an mHealth app when reporting personal issues than in their physician [73]. Anonymity helps overcome thoughts of stigmatization or shame when describing personal issues [74].

We are also missing implementations of psychological theories in CBT mental health apps to increase the impact of and adherence to these technological means. Peters et al [75] suggested a framework grounded in psychological research that could help developers of mHealth apps develop apps that increase motivation and engagement and, in this way, also have an impact on patients’ well-being.

Recommendation of Apps

Our literature search showed that the underlying clinical evidence, technical issues, and implementation details are rarely described in the published research on mental health apps. Without information on the correctness of the underlying evidence base and without confirmation of efficacy, it is challenging for patients to identify the mental health apps that are suited for them.

Therapists and patients rely on lists of top mental health apps for their decision-making process, which are not helpful unless they list their ranking and selection criteria for creating the list. We checked for 21 apps listed on the mental health app ranking lists to determine whether there were publications available. A scientific paper or results from a clinical trial were available for only 29% (6/21) of the apps. This makes it difficult for users and health professionals to identify high-quality, evidence-based apps. It must be ensured that the underlying principles and delivered contents are evidence based [17,76] to avoid harm to users, as well as undesired consequences for the users. The number of downloads from an app store is clearly not an indicator of high-quality mHealth apps. On the other hand, we found studies on 28 apps, of which only 16 (57%) were implemented for the established Android and iOS operating systems. Of these 16 apps, only 12 (75%) were available in the stores. This means that apps that have been thoroughly investigated are not available to a wide range of users.

We conclude that there is a need to enable patients in informed decision-making when selecting an mHealth app for CBT [77]. We suggest a minimal data set of information on a mental health app that includes the following items: underlying clinical evidence and CBT techniques that are integrated; information on how CBT techniques are implemented in the app, that is, which technologies are used and for which purpose; efficacy from an RCT, if available; intended application areas of the app; possible contraindications; information on data storage, data security, and privacy; and the integrated third-party tools.

Standardized evaluation metrics should be developed to ensure that only high-quality apps are recommended by therapists. Attempts have been made to harmonize technical metrics for evaluating health chatbots [78,79]. Existing evaluation frameworks for mHealth apps target usability [76]. Although good usability is essential to make sure that users can interact with the app, it is highly relevant that the app really positively affects patients and does not harm them.

Strengths and Limitations of Our Presented Research Work

Strengths

This literature review was conducted to identify the technologies used in mHealth apps for delivering CBT and identify the research gaps. To the best of our knowledge, this study is among the first to summarize these aspects and provide an overview. This study helps in identifying the opportunities and limitations of mHealth in supporting CBT. It also shows the potential and limitations of e–mental health with respect to the self-help capabilities of patients.

As 2 reviewers independently selected the articles and extracted the data, selection bias in this review was reduced. There is a review of Stawarz et al [80] that studies user experience of CBT-enabling apps [80]. In contrast to their work, we are more interested in the technical implementation of CBT in mHealth apps and the research gaps and not only in the features provided by the apps.

Limitations

The search in this review was restricted to English and German articles. Accordingly, it is likely that this review missed some publications. We also missed apps for which no publications were available. In addition, we restricted the search to papers that included the terms cognitive behavior therapy or mHealth or the respective Medical Subject Headings terms. We might have missed relevant papers that did not explicitly use these terms but dealt with a relevant topic. Several terms can be related to mHealth; however, these are too broad to be reviewed in a reasonable amount of time.

Conclusions

In this paper, we studied the therapeutic aspects of CBT that are implemented in mHealth apps and the technologies by which they are integrated into these apps. We conclude that some CBT techniques (behavioral activation and cognitive restructuring) can be well-realized in a mental health app, whereas others are more difficult to implement (problem solving and exposure). mHealth apps for CBT can support patients through additional self-help and self-management tools that support specific aspects of treatment. mHealth apps facilitate the self-competence and self‑management of patients in coping with mental health problems. Interactive elements, gamification, and integration of social networks are technologies that increase user engagement and adherence. In this way, patients are provided with additional action and interaction capabilities. As they cover relevant aspects of CBT treatment, mHealth apps can, in principle, alleviate the issue of a global shortage of mental health human resources, as they are usually available to millions of users anytime and anywhere, presuming the users have mobile devices and internet access. This, in turn, has the potential to improve the availability and quality of mental health care at reduced expenses. However, these apps must be evidence based and reliable to be effective and avoid harm to patients. Further research is needed on the side effects of mHealth self-help treatments, such as new inequalities because of the digital divide, addictive internet behavior, trust in mHealth, anonymity issues, risk and range of self-help treatments, bias toward user groups and social contexts, and ethical implications.

The future role of app-based e–mental health still has to be clarified. Future research should assess integrated care models where all stakeholders can benefit: the health care provider by avoiding repetitive tasks and the patient by receiving support even beyond therapy sessions. In addition, economic value is relevant for bringing evidence-based apps to users and supporting clinical care. For obsessive-compulsive disorders, it has already been shown that computerized CBT delivered as low-intensity interventions is cost-effective [81].

Conflicts of Interest

None declared.

Multimedia Appendix 1

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

DOCX File , 18 KB

Multimedia Appendix 2

Study designs reported in the papers.

DOCX File , 13 KB

Multimedia Appendix 3

Cognitive behavioral therapy tools and methods realized in the reported apps.

DOCX File , 14 KB

Multimedia Appendix 4

Technologies used in the apps.

DOCX File , 13 KB

  1. Steel Z, Marnane C, Iranpour C, Chey T, Jackson JW, Patel V, et al. The global prevalence of common mental disorders: a systematic review and meta-analysis 1980-2013. Int J Epidemiol 2014;43(2):476-493 [FREE Full text] [CrossRef] [Medline]
  2. Whiteford HA, Ferrari AJ, Degenhardt L, Feigin V, Vos T. The global burden of mental, neurological and substance use disorders: an analysis from the Global Burden of Disease Study 2010. PLoS One 2015;10(2):e0116820 [FREE Full text] [CrossRef] [Medline]
  3. Angermeyer MC, Matschinger H. The stigma of mental illness: effects of labelling on public attitudes towards people with mental disorder. Acta Psychiatr Scand 2003;108(4):304-309. [CrossRef] [Medline]
  4. Cuijpers P, Sijbrandij M, Koole SL, Andersson G, Beekman AT, Reynolds 3rd CF. The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisons. World Psychiatry 2013;12(2):137-148 [FREE Full text] [CrossRef] [Medline]
  5. Murray CJ, Vos T, Lozano R, Naghavi M, Flaxman AD, Michaud C, et al. Disability-adjusted life years (DALYs) for 291 diseases and injuries in 21 regions, 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet 2012;380(9859):2197-2223. [CrossRef] [Medline]
  6. Oladeji BD, Gureje O. Brain drain: a challenge to global mental health. BJPsych Int 2016;13(3):61-63 [FREE Full text] [CrossRef] [Medline]
  7. Sadeniemi M, Almeda N, Salinas-Pérez JA, Gutiérrez-Colosía MR, García-Alonso C, Ala-Nikkola T, et al. A comparison of mental health care systems in northern and southern Europe: a service mapping study. Int J Environ Res Public Health 2018;15(6):1133 [FREE Full text] [CrossRef] [Medline]
  8. Anthes E. Mental health: there's an app for that. Nature 2016;532(7597):20-23. [CrossRef] [Medline]
  9. Hester RD. Lack of access to mental health services contributing to the high suicide rates among veterans. Int J Ment Health Syst 2017;11:47 [FREE Full text] [CrossRef] [Medline]
  10. MoodGym.   URL: https://moodgym.com.au, [accessed 2021-02-04]
  11. Christensen H, Griffiths KM, Evans K. e-mental health in Australia: implications of the internet and related technologies for policy. Canberra, Australia: Commonwealth Department of Health and Ageing; 2002.
  12. Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry 2014;13(3):288-295 [FREE Full text] [CrossRef] [Medline]
  13. Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlöf E. Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther 2018;47(1):1-18. [CrossRef] [Medline]
  14. Moock J. Support from the internet for individuals with mental disorders: advantages and disadvantages of e-mental health service delivery. Front Public Health 2014;2:65 [FREE Full text] [CrossRef] [Medline]
  15. Wenzel A. Basic strategies of cognitive behavioral therapy. Psychiatr Clin North Am 2017;40(4):597-609 [FREE Full text] [CrossRef] [Medline]
  16. Chandrashekar P. Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth 2018;4:6 [FREE Full text] [CrossRef] [Medline]
  17. Bendig E, Erb B, Schulze-Thuesing L, Baumeister H. The next generation: chatbots in clinical psychology and psychotherapy to foster mental health – a scoping review. Verhaltenstherapie 2019:1-13 [FREE Full text] [CrossRef]
  18. Huppert JD. The building blocks of treatment in cognitive-behavioral therapy. Isr J Psychiatry Relat Sci 2009;46(4):245-250 [FREE Full text] [Medline]
  19. Butler AC, Chapman JE, Forman EM, Beck AT. The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clin Psychol Rev 2006;26(1):17-31. [CrossRef] [Medline]
  20. Reckwitz A. Toward a theory of social practices. Eur J Soc Theory 2002;5(2):243-263 [FREE Full text] [CrossRef]
  21. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 2009;62(10):1006-1012. [CrossRef] [Medline]
  22. Stefanovics EA, Potenza MN, Pietrzak RH. PTSD and obesity in U.S. military veterans: prevalence, health burden, and suicidality. Psychiatry Res 2020;291:113242. [CrossRef] [Medline]
  23. Martens M. Issues of access and usability in designing digital resources for children. Libr Inf Sci Res 2012;34(3):159-168 [FREE Full text] [CrossRef]
  24. Plaza García I, Sánchez CM, Espílez ÁS, García-Magariño I, Guillén GA, García-Campayo J. Development and initial evaluation of a mobile application to help with mindfulness training and practice. Int J Med Inform 2017;105:59-67. [CrossRef] [Medline]
  25. Denecke K, Vaaheesan S, Arulnathan A. A mental health chatbot for regulating emotions (SERMO) - concept and usability test. IEEE Trans Emerg Topics Comput 2021;9(3):1170-1182 [FREE Full text] [CrossRef]
  26. Purkayastha S, Addepally SA, Bucher S. Engagement and usability of a cognitive behavioral therapy mobile app compared with web-based cognitive behavioral therapy among college students: randomized heuristic trial. JMIR Hum Factors 2020;7(1):e14146 [FREE Full text] [CrossRef] [Medline]
  27. Martínez-Miranda J, Martínez A, Ramos R, Aguilar H, Jiménez L, Arias H, et al. Assessment of users' acceptability of a mobile-based embodied conversational agent for the prevention and detection of suicidal behaviour. J Med Syst 2019;43(8):246. [CrossRef] [Medline]
  28. Furukawa TA, Imai H, Horikoshi M, Shimodera S, Hiroe T, Funayama T, FLATT Investigators. Behavioral activation: is it the expectation or achievement, of mastery or pleasure that contributes to improvement in depression? J Affect Disord 2018;238:336-341. [CrossRef] [Medline]
  29. Bakker D, Kazantzis N, Rickwood D, Rickard N. A randomized controlled trial of three smartphone apps for enhancing public mental health. Behav Res Ther 2018;109:75-83. [CrossRef] [Medline]
  30. Mantani A, Kato T, Furukawa TA, Horikoshi M, Imai H, Hiroe T, et al. Smartphone cognitive behavioral therapy as an adjunct to pharmacotherapy for refractory depression: randomized controlled trial. J Med Internet Res 2017;19(11):e373 [FREE Full text] [CrossRef] [Medline]
  31. Furukawa TA, Horikoshi M, Fujita H, Tsujino N, Jinnin R, Kako Y, et al. Cognitive and behavioral skills exercises completed by patients with major depression during smartphone cognitive behavioral therapy: secondary analysis of a randomized controlled trial. JMIR Ment Health 2018;5(1):e4 [FREE Full text] [CrossRef] [Medline]
  32. Monney G, Penzenstadler L, Dupraz O, Etter JF, Khazaal Y. mHealth app for cannabis users: satisfaction and perceived usefulness. Front Psychiatry 2015;6:120 [FREE Full text] [CrossRef] [Medline]
  33. Christoforou M, Sáez Fonseca JA, Tsakanikos E. Two novel cognitive behavioral therapy-based mobile apps for agoraphobia: randomized controlled trial. J Med Internet Res 2017;19(11):e398 [FREE Full text] [CrossRef] [Medline]
  34. Depp CA, Perivoliotis D, Holden J, Dorr J, Granholm EL. Single-session mobile-augmented intervention in serious mental illness: a three-arm randomized controlled trial. Schizophr Bull 2019;45(4):752-762 [FREE Full text] [CrossRef] [Medline]
  35. Prada P, Zamberg I, Bouillault G, Jimenez N, Zimmermann J, Hasler R, et al. EMOTEO: a smartphone application for monitoring and reducing aversive tension in borderline personality disorder patients, a pilot study. Perspect Psychiatr Care 2017;53(4):289-298. [CrossRef] [Medline]
  36. Rohani DA, Lopategui AQ, Tuxen N, Faurholt-Jepsen M, Kessing LV, Bardram JE. MUBS: a personalized recommender system for behavioral activation in mental health. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 2020 Presented at: CHI '20; April 25-30, 2020; Honolulu, HI p. 1-13. [CrossRef]
  37. Chen YX, Chen HC, Chen LX, Hu JW, Shie CK, Lin YS, et al. Enhancing adherence to cognitive behavioral therapy for insomnia through machine and social persuasion. In: IEEE International Conference on Green Computing and Communications and IEEE Internet of Things and IEEE Cyber, Physical and Social Computing. 2013 Presented at: CPSCom & GreenCom '13; August 20-23, 2013; Beijing, China p. 750-757   URL: https://doi.org/10.1109/greencom-ithings-cpscom.2013.135 [CrossRef]
  38. O'Connor M, Whelan R, Bricker J, McHugh L. Randomized controlled trial of a smartphone application as an adjunct to acceptance and commitment therapy for smoking cessation. Behav Ther 2020;51(1):162-177. [CrossRef] [Medline]
  39. Nobis S, Lehr D, Ebert DD, Berking M, Heber E, Baumeister H, et al. Efficacy and cost-effectiveness of a web-based intervention with mobile phone support to treat depressive symptoms in adults with diabetes mellitus type 1 and type 2: design of a randomised controlled trial. BMC Psychiatry 2013;13:306 [FREE Full text] [CrossRef] [Medline]
  40. Scherr S, Goering M. Is a self-monitoring app for depression a good place for additional mental health information? Ecological momentary assessment of mental help information seeking among smartphone users. Health Commun 2020;35(8):1004-1012. [CrossRef] [Medline]
  41. Rohani D, Tuxen N, Lopategui AQ, Faurholt-Jepsen M, Kessing LV, Bardram JE. Personalizing mental health: a feasibility study of a mobile behavioral activation tool for depressed patients. In: Proceedings of the 13th EAI International Conference on Pervasive Computing Technologies for Healthcare. 2019 Presented at: PervasiveHealth '19; May 20-23, 2019; Trento, Italy p. 282-291. [CrossRef]
  42. Bucci S, Barrowclough C, Ainsworth J, Machin M, Morris R, Berry K, et al. Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophr Bull 2018;44(5):1070-1080 [FREE Full text] [CrossRef] [Medline]
  43. Anastasiadou D, Folkvord F, Brugnera A, Cañas Vinader L, SerranoTroncoso E, Carretero Jardí C, et al. An mHealth intervention for the treatment of patients with an eating disorder: a multicenter randomized controlled trial. Int J Eat Disord 2020;53(7):1120-1131. [CrossRef] [Medline]
  44. Tudor-Sfetea C, Rabee R, Najim M, Amin N, Chadha M, Jain M, et al. Evaluation of two mobile health apps in the context of smoking cessation: qualitative study of cognitive behavioral therapy (CBT) versus non-CBT-based digital solutions. JMIR Mhealth Uhealth 2018;6(4):e98 [FREE Full text] [CrossRef] [Medline]
  45. Clarke J, Proudfoot J, Birch MR, Whitton AE, Parker G, Manicavasagar V, et al. Effects of mental health self-efficacy on outcomes of a mobile phone and web intervention for mild-to-moderate depression, anxiety and stress: secondary analysis of a randomised controlled trial. BMC Psychiatry 2014;14:272 [FREE Full text] [CrossRef] [Medline]
  46. Moberg C, Niles A, Beermann D. Guided self-help works: randomized waitlist controlled trial of pacifica, a mobile app integrating cognitive behavioral therapy and mindfulness for stress, anxiety, and depression. J Med Internet Res 2019;21(6):e12556 [FREE Full text] [CrossRef] [Medline]
  47. Bardram JE, Rohani DA, Tuxen N, Faurholt-Jepsen M, Kessing LV. Supporting smartphone-based behavioral activation: a simulation study. In: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. 2017 Presented at: UbiComp '17; September 11-15, 2017; Maui, HI p. 830-843. [CrossRef]
  48. Champion L, Economides M, Chandler C. The efficacy of a brief app-based mindfulness intervention on psychosocial outcomes in healthy adults: a pilot randomised controlled trial. PLoS One 2018;13(12):e0209482 [FREE Full text] [CrossRef] [Medline]
  49. Nobis S, Ebert DD, Lehr D, Smit F, Buntrock C, Berking M, et al. Web-based intervention for depressive symptoms in adults with types 1 and 2 diabetes mellitus: a health economic evaluation. Br J Psychiatry 2018;212(4):199-206. [CrossRef] [Medline]
  50. Huberty J, Green J, Glissmann C, Larkey L, Puzia M, Lee C. Efficacy of the mindfulness meditation mobile app "Calm" to reduce stress among college students: randomized controlled trial. JMIR Mhealth Uhealth 2019;7(6):e14273 [FREE Full text] [CrossRef] [Medline]
  51. Tregarthen J, Paik Kim J, Sadeh-Sharvit S, Neri E, Welch H, Lock J. Comparing a tailored self-help mobile app with a standard self-monitoring app for the treatment of eating disorder symptoms: randomized controlled trial. JMIR Ment Health 2019;6(11):e14972 [FREE Full text] [CrossRef] [Medline]
  52. Kim SW, Lee GY, Yu HY, Jung EI, Lee JY, Kim SY, et al. Development and feasibility of smartphone application for cognitive-behavioural case management of individuals with early psychosis. Early Interv Psychiatry 2018;12(6):1087-1093. [CrossRef] [Medline]
  53. Arean PA, Hallgren KA, Jordan JT, Gazzaley A, Atkins DC, Heagerty PJ, et al. The use and effectiveness of mobile apps for depression: results from a fully remote clinical trial. J Med Internet Res 2016;18(12):e330 [FREE Full text] [CrossRef] [Medline]
  54. Cheng P, Kalmbach DA, Tallent G, Joseph CL, Espie CA, Drake CL. Depression prevention via digital cognitive behavioral therapy for insomnia: a randomized controlled trial. Sleep 2019;42(10):zsz150 [FREE Full text] [CrossRef] [Medline]
  55. Edmunds JM, Brodman DM, Ringle VA, Read KL, Kendall PC, Beidas RS. Examining adherence to components of cognitive-behavioral therapy for youth anxiety after training and consultation. Prof Psychol Res Pr 2017;48(1):54-61 [FREE Full text] [CrossRef] [Medline]
  56. Khademian F, Aslani A, Bastani P. The effects of mobile apps on stress, anxiety, and depression: overview of systematic reviews. Int J Technol Assess Health Care 2020;37:e4. [CrossRef] [Medline]
  57. Martinengo L, Van Galen L, Lum E, Kowalski M, Subramaniam M, Car J. Suicide prevention and depression apps' suicide risk assessment and management: a systematic assessment of adherence to clinical guidelines. BMC Med 2019;17(1):231 [FREE Full text] [CrossRef] [Medline]
  58. Dimidjian S, Barrera Jr M, Martell C, Muñoz RF, Lewinsohn PM. The origins and current status of behavioral activation treatments for depression. Annu Rev Clin Psychol 2011;7:1-38. [CrossRef] [Medline]
  59. Morriss RK, Faizal MA, Jones AP, Williamson PR, Bolton C, McCarthy JP. Interventions for helping people recognise early signs of recurrence in bipolar disorder. Cochrane Database Syst Rev 2007;2007(1):CD004854 [FREE Full text] [CrossRef] [Medline]
  60. East ML, Havard BC. Mental health mobile apps: from infusion to diffusion in the mental health social system. JMIR Ment Health 2015;2(1):e10 [FREE Full text] [CrossRef] [Medline]
  61. Meskó B, Drobni Z, Bényei É, Gergely B, Győrffy Z. Digital health is a cultural transformation of traditional healthcare. Mhealth 2017;3:38 [FREE Full text] [CrossRef] [Medline]
  62. Larsen ME, Nicholas J, Christensen H. A systematic assessment of smartphone tools for suicide prevention. PLoS One 2016;11(4):e0152285 [FREE Full text] [CrossRef] [Medline]
  63. Bentz D, Wang N, Ibach MK, Schicktanz NS, Zimmer A, Papassotiropoulos A, et al. Effectiveness of a stand-alone, smartphone-based virtual reality exposure app to reduce fear of heights in real-life: a randomized trial. NPJ Digit Med 2021;4(1):16 [FREE Full text] [CrossRef] [Medline]
  64. Pramana G, Parmanto B, Lomas J, Lindhiem O, Kendall PC, Silk J. Using mobile health gamification to facilitate cognitive behavioral therapy skills practice in child anxiety treatment: open clinical trial. JMIR Serious Games 2018;6(2):e9 [FREE Full text] [CrossRef] [Medline]
  65. D'Alfonso S. AI in mental health. Curr Opin Psychol 2020;36:112-117. [CrossRef] [Medline]
  66. Vaidyam AN, Wisniewski H, Halamka JD, Kashavan MS, Torous JB. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry 2019;64(7):456-464 [FREE Full text] [CrossRef] [Medline]
  67. Hariton E, Locascio JJ. Randomised controlled trials - the gold standard for effectiveness research: study design: randomised controlled trials. BJOG 2018;125(13):1716 [FREE Full text] [CrossRef] [Medline]
  68. Bauer M, Glenn T, Geddes J, Gitlin M, Grof P, Kessing LV, et al. Smartphones in mental health: a critical review of background issues, current status and future concerns. Int J Bipolar Disord 2020;8(1):2 [FREE Full text] [CrossRef] [Medline]
  69. Neff G, Nagy P. Agency in the digital age: using symbiotic agency to explain human-technology interaction. In: Papacharissi Z, editor. A networked self and human augmentics, artificial intelligence, sentience. New York, NY: Routledge; 2018:97-107.
  70. Armstrong D. Actors, patients and agency: a recent history. Sociol Health Illn 2014;36(2):163-174. [CrossRef] [Medline]
  71. Kretzschmar K, Tyroll H, Pavarini G, Manzini A, Singh I, NeurOx Young People’s Advisory Group. Can your phone be your therapist? Young people's ethical perspectives on the use of fully automated conversational agents (Chatbots) in mental health support. Biomed Inform Insights 2019;11:1178222619829083 [FREE Full text] [CrossRef] [Medline]
  72. Elhai JD, Dvorak RD, Levine JC, Hall BJ. Problematic smartphone use: a conceptual overview and systematic review of relations with anxiety and depression psychopathology. J Affect Disord 2017;207:251-259. [CrossRef] [Medline]
  73. Denecke K, Hochreutener SL, Pöpel A, May R. Self-anamnesis with a conversational user interface: concept and usability study. Methods Inf Med 2018;57(5-06):243-252. [CrossRef] [Medline]
  74. Elias J. In 2016, users will trust health apps more than their doctors. Forbes. 2015.   URL: http:/​/www.​forbes.com/​sites/​jenniferelias/​2015/​12/​31/​in-2016-users-will-trust-health-apps-more-than-their-doctors/​#4a244142d5f6 [accessed 2021-11-30]
  75. Peters D, Calvo RA, Ryan RM. Designing for motivation, engagement and wellbeing in digital experience. Front Psychol 2018;9:797 [FREE Full text] [CrossRef] [Medline]
  76. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3(1):e27 [FREE Full text] [CrossRef] [Medline]
  77. Lipschitz J, Miller CJ, Hogan TP, Burdick KE, Lippin-Foster R, Simon SR, et al. Adoption of mobile apps for depression and anxiety: cross-sectional survey study on patient interest and barriers to engagement. JMIR Ment Health 2019;6(1):e11334 [FREE Full text] [CrossRef] [Medline]
  78. Abd-Alrazaq A, Safi Z, Alajlani M, Warren J, Househ M, Denecke K. Technical metrics used to evaluate health care chatbots: scoping review. J Med Internet Res 2020;22(6):e18301 [FREE Full text] [CrossRef] [Medline]
  79. Denecke K, Warren J. How to evaluate health applications with conversational user interface? Stud Health Technol Inform 2020;270:976-980. [CrossRef] [Medline]
  80. Stawarz K, Preist C, Tallon D, Wiles N, Coyle D. User experience of cognitive behavioral therapy apps for depression: an analysis of app functionality and user reviews. J Med Internet Res 2018;20(6):e10120 [FREE Full text] [CrossRef] [Medline]
  81. Lovell K, Bower P, Gellatly J, Byford S, Bee P, McMillan D, et al. Clinical effectiveness, cost-effectiveness and acceptability of low-intensity interventions in the management of obsessive-compulsive disorder: the Obsessive-Compulsive Treatment Efficacy randomised controlled Trial (OCTET). Health Technol Assess 2017;21(37):1-132 [FREE Full text] [CrossRef] [Medline]


CBT: cognitive behavioral therapy
mHealth: mobile health
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
RCT: randomized controlled trial
WHO: World Health Organization


Edited by A Mavragani; submitted 06.02.21; peer-reviewed by Y Khazaal, R Calvo, S Ng; comments to author 02.06.21; revised version received 27.07.21; accepted 28.12.21; published 10.03.22

Copyright

©Kerstin Denecke, Nicole Schmid, Stephan Nüssli. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 10.03.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.