Internet-delivered cognitive behavioral therapy (iCBT) for depression and anxiety has been developed to help increase access to evidence-based therapies. There is empirical support for its use in treating depression and anxiety [1-4]. End users experience it positively [5] and find it to be satisfactory and acceptable [1,6,7]. However, disseminating iCBT at scale remains a challenge [8,9], and COVID-19 has brought its relevance to light now more so than ever [10-12]. A 2019 commentary [13] discussed the evidence-to-practice gap in digital mental health treatments. The authors postulated that the reason for this gap is a lack of knowledge in the field of iCBT around implementing interventions within routine care. They suggested the adoption of implementation science methodologies to bridge this evidence-to-practice gap.

Implementation science has been defined as the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice to improve the quality and effectiveness of health services and care [14]. Central to this definition is the problem statement behind it: it takes almost 17 years for health care research to achieve its intended benefit, which is termed as the “evidence-to-practice gap” [15,16]. As a newly emerging academic field, implementation science is largely integrative; it borrows and adapts theories from multiple fields and uses these to understand the determinant mechanisms as to why (or why not) a specific implementation succeeds [17]. Implementation science theories provide a framework that allows for implementation plans to be developed and relevant outcomes to be measured [18], and it has been posited that using these methodologies in future studies of iCBT could generate learnings relevant to its real-world application [13,19,20].

In a recent review focusing on determinants of implementation for eHealth interventions [8], 37 determinants associated with successful implementation were identified. However, it should be noted that “eHealth interventions” in this case contained a wide variety of digitally enabled interventions, including iCBT and psychotherapy delivered via videoconferencing. When comparing iCBT and other eHealth interventions, “complexity” is a factor for consideration, that is, the degree to which an intervention contains multiple components that require interaction from many individuals, from various levels within an organization, to enact the intervention effectively [21]. iCBT’s level of complexity is highlighted in service illustration articles by Titov et al [22,23]; for example, therapists’ skill set to operate iCBT efficiently (technical knowledge and constructing written messages), revised services delivery pathways, adherence to regulatory frameworks, and newly aligned clinical governance procedures are some elements of how delivering iCBT may differ from more traditional or less complex services. Conversely, although administering psychological therapy through videoconferencing software may require some altering of specific therapeutic skills and technical upskilling [24], relative complexity across other areas may be lower (eg, referral pathways and wider system integration). Similarly, some authors have illustrated the need for both iCBT [25,26] and telehealth-specific competency frameworks [27], further illustrating the need for specialized skills to extend the traditional therapist skill set.

Attempts to mobilize implementation science information on eHealth interventions, generally to the point of having pragmatic, clinical relevance for iCBT, have been sparse [28]. As a consequence, the availability of implementation findings relevant to iCBT remains low. This study is a mixed methods systematic review that aimed to account for literature that specifically references or can inform factors relevant to the implementation of iCBT, specifically for depression and anxiety treatment in adults. The benefit of a mixed methods systematic review over traditional systematic literature reviewing is that it seeks to extract relevant information across qualitative, quantitative, review, and illustration-based articles. Mixed methods synthesis affords a way to effectively capture this information and synthesize it qualitatively to produce relevant insights into the implementation of iCBT. Therefore, a convergent integrated approach to the mixed methods review was chosen due to its appropriateness over other review methods for the subject; a traditional systematic review on the implementation of iCBT for depression and anxiety would not be appropriate due to insufficient qualitative or quantitative findings to generate insights [29]. Furthermore, there are no restrictions imposed on the type of evidence included within the synthesis, which aligns with the anticipated variety of articles that would be identified [30]. The disorder domains of depression and anxiety were chosen due to them being the most substantive areas of research for iCBT.

A mixed methods systematic review departs from and complements previous work in the ways mentioned subsequently [8]. First, it will specifically focus on iCBT-based interventions, which can be considered relatively “complex” in their administration [21]. Second, it will provide a rich description of the current “practice behind the science” by focusing on reportage within the methods, results, and discussion sections of the articles. Third, it will contribute to the existing literature regarding specific implementation strategies that are associated with the use of iCBT [31]. Finally, it will allow the interpretation of research findings in a way that is hoped to be productive for future implementations specific to iCBT for the treatment of depression and anxiety.

The overarching objective of the review was centered on the pragmatic question of “What can we learn from published peer-reviewed literature about the implementation of iCBT for depression and anxiety?” This objective was further broken down into 2 domains of focus, on which data extraction and subsequent data analysis were based (refer to the Methods section).

The first domain was centered on implementation insights derived from iCBT research. This objective and domain center on understanding the novel information that is often presented in published research and how this information can have relevance and be mobilized for the benefit of iCBT implementation.

The second domain was implementation process considerations for the successful implementation of iCBT, which consisted of establishing the strategies that are used within the literature to facilitate the implementation of iCBT. According to implementation science literature, implementation strategies are methods used to facilitate the implementation of an intervention, where strategies can consist of training packages, management approaches, developing protocols for intervention use, etc [31,32].

Because implementation science information on eHealth interventions that has pragmatic, clinical relevance for iCBT has been sparse and scattered across different article types, a mixed methods systematic review was conducted. This mixed methods systematic review used a convergent integrated approach and was conducted to identify literature that was central to the review objective mentioned earlier [30,33,34]. The convergent integrated approach to conducting a mixed methods systematic review consists of “qualitizing” numerical or statistical findings; that is, quantitative findings are extracted and allocated textual descriptions to allow for integration and simultaneous synthesis with other qualitative data. The resulting qualitative data were then analyzed using the descriptive-interpretive approach [35]. This review was not registered, and a review protocol was not prepared.

The general search strategy used was as follows: (“ICBT” OR “CCBT” OR “internet-delivered CBT” OR “internet-delivered cognitive behavioural therapy” OR “internet-delivered cognitive behaviour therapy” OR “internet-based cognitive behaviour therapy” OR “internet-based cognitive behavioural therapy” OR “internet-administered cognitive behaviour therapy” OR “internet-administered cognitive behavioural therapy” OR “internet-supported cognitive behaviour therapy” OR “internet-supported cognitive behavioural therapy”) AND (“Anx*” OR “depress*” OR “low mood” OR “GAD” OR “phobia” OR “SAD”) AND “Implement*.” Databases searched included PsycINFO, PsycArticles, MEDLINE, CINAHL Complete, and Embase. A full description of terms and derivatives for the different databases is included in Multimedia Appendix 1. Search engine limitations required that the search date had to start from 2007. The search was initially conducted in June 2020 (January 1, 2007-June 1, 2020) and further repeated in September 2021 (June 1, 2020-August 31, 2021) to identify any new or relevant publications. The inclusion and exclusion criteria are given in Textbox 1.

The screening consisted of two steps: (1) review the title and abstract and (2) review the full-paper. We chose to first review the titles and abstracts of all identified records due to the nature of this review and the wide range of study types that were anticipated to result from the search. For example, it was noted throughout the reviewing process that articles frequently cited the terms “implementation” or “feasibility” in the title but failed to provide any relevant information under these constructs when abstracts were reviewed. Where articles provided inadequate information in their abstracts (eg, “the results inform the feasibility of implementing iCBT within XYZ context”) to apply the inclusion criteria, DR acted as a second reviewer for these abstracts and consulted with DD (primary reviewer) to make a decision on inclusion or exclusion. Once step 1 was completed, full texts of all articles were reviewed by DD at full text to discern their relevance to the domains of interest. During this review, articles were rejected during full text screening for the following two reasons: (1) incorrect record specification from the databases (eg, conference presentations being mislabeled) and (2) provided little (eg, minor comments relating to the implementation of iCBT within “future research” sections) or no information on implementation. Once all articles were screened and the final dataset established, data extraction commenced.

A total of 40 eligible articles published between 2010 and 2021 were included in the mixed methods synthesis. Multimedia Appendix 2 provides the types of articles included in this mixed methods systematic review and brief summaries of these included articles. Article types included in this review were case study (1/40, 2%), commentary (4/40, 10%), meta-analysis (5/40, 12%), mixed methods study (3/40, 8%), pilot randomized controlled trial (1/40, 2%), randomized controlled trial (2/40, 5%), qualitative study (5/40, 12%), quantitative study (13/40, 32%), review (3/40, 8%), and systematic review (3/40, 8%). Multimedia Appendix 3 [4,9,22,23,28,37-71] includes a numbered list of all articles analyzed as part of this mixed methods systematic review. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram constructed to illustrate the search findings is illustrated in Figure 1 and the PRISMA checklist is provided in Multimedia Appendix 4.

Two domains (and lower subdomains, categories, and subcategories) were identified: (1) implementation insights derived from iCBT research and (2) implementation processes related to the successful implementation of iCBT in care settings (Tables 1 and 2). Subcategories are discussed as talking points within each category as opposed to adding further organizational levels within this study.

Our mixed methods systematic review highlights the knowledge we have gained from the available literature on the implementation of iCBT. Some of our key findings regarding the process for implementing iCBT include the practice of iCBT, with special reference to determining client eligibility and effectively supporting patients in iCBT. The management of iCBT in the workplace, especially staff and operational considerations, also surfaced as an important process to consider when implementing. Other related findings include the importance of staff training, the management of treatment pathways, security, and factors for consideration within the wider context that impact the implementation of iCBT. In terms of implementation insights, this review has highlighted that clinician and patient attitudes toward iCBT can influence its ability to achieve intended outcomes and the need to continually tailor iCBT for patient benefit and that further research can help to develop our understanding of implementing iCBT successfully.

As would be expected, the practice of iCBT was highlighted as important to the implementation process for iCBT within a mental health service. For instance, what constitutes eligibility for an iCBT intervention manifested in two categories: (1) screening and inclusion criteria for accessing iCBT need to be thoroughly defined and (2) consideration of the usefulness of iCBT beyond the original target of mild-to-moderate depression and anxiety. Historically, eligibility for iCBT has been characterized by low symptom presence (mild to moderate) and no significant risk issues. This approach was sensible while establishing iCBT’s safety and effectiveness as an intervention, subsequently resulting in a well-validated evidence base supporting iCBT for treating depression and anxiety. Consequently, the preponderance of historical eligibility seems to be an artifact in need of revision. This is particularly important in light of the growing body of literature to support iCBT’s applicability to more severe presentations of mental health difficulties [72-75]. In addition, real-world data from iCBT clinics highlight that a substantial proportion of patients accessing these services have presentations within the moderate to severe range [22].

There is a clear need for services to consider the populations they serve (eg, general severity levels and client demographics) and tailor their model of iCBT provision. Still, despite the available evidence, clinical guidelines lag in their support for iCBT in extended service delivery pathways [76]. This situation poses some difficulty for certain services or health systems to innovate their use of digital interventions (eg, the improving access to psychological therapies program in England, which offers treatment based on National Institute for Health and Care Excellence guidelines). Specifically in the English context, the original guideline for the use of iCBT was rolled out in 2004 [77,78] and was updated in 2022. Since then, technologies and research have developed and would suggest the utility of iCBT for the broader population.

Our results highlight the importance of the operational aspects of iCBT services. First, the importance of effective management and leadership to support the implementation of iCBT was identified through the review. Transformational leadership approaches, that is, leadership styles associated with motivating and compelling employees to participate in a shared vision [79], have been found to be associated with increased levels of innovation climate, further defined as an organizational climate that is conducive to the adoption of novel, evidence-based practices [80]. Implementing iCBT requires leaders to navigate interactions across multiple levels of a service and motivate staff to ensure the vision of iCBT is fulfilled. However, the current studies identified do not illustrate in depth the effects of leadership nor was it their primary or secondary focus. Despite this, it is still important that this finding was communicated through this small pool of studies, and more research is needed to inform this gap in knowledge.

Training staff in iCBT and increasing their motivation to use it were both cited as important. As an in-service activity, training clinicians and therapists in the use of evidence-based practice has a substantial literature base [81,82]. However, our findings highlight variance in the training delivered to therapists charged with delivering iCBT, ranging from hours to up to a year of continued education, and the components of the training were also not described at length across articles. The wider literature on training stands in contrast to what we identified; training programs for evidence-based practice tend to produce better outcomes (eg, competency, evidence-based practice use, and positive attitudes) when multicomponent approaches are used (eg, workshop, follow-up, and audit of skills acquired) [81]. To date, no systematic evaluation of iCBT training programs has been conducted, and it has also been cited by one of the included studies that training programs for these interventions are rare [37]. Similarly, we identified that staff motivation to use iCBT needs to be fostered. This motivation can also be developed through training initiatives, where implementers can illustrate the benefits that iCBT brings to routine clinical practice (eg, improves patient symptoms and access to care, is usable, and is not time consuming), and this activity may potentially influence motivation around intervention use [83,84].

Furthermore, routine monitoring of the intervention and its outcomes was also cited as important for the continued development of iCBT within the service. This activity can allow supporters in iCBT to reflect on their own practice to improve service provision, with an article stating that clinicians who administer iCBT desire comprehensive updates regarding iCBT to understand its impact on wider service outcomes [28]. This activity is reminiscent of the construct “reflexive monitoring” from normalization process theory [84], where individual and group reflections on processes around a specific evidence-based practice can lead to revisions in practice that are adapted to best suit the needs and structures of the service context. The results regarding the operational aspects of iCBT, despite not being widely reported across the literature, indicate that factors associated with evidence-based practice success in the implementation literature are being considered when iCBT is implemented, which is a promising finding. More widespread reporting of this information could be beneficial to practicing professionals when making choices about using iCBT with their patients.

Patients tend to be positive about iCBT and the support they receive, but clinician attitudes generally lean toward the negative. From service illustrations, we can infer that clinicians receive significant exposure to iCBT when it is implemented [22,23,38], and its effectiveness is grounded in the literature. Negative attitudes can result in the abandonment of the implementation effort due to a lack of acceptance or misunderstandings around the perceived value of the treatment [85,86]. A better understanding of negative clinician attitudes can be attained if iCBT were to be interpreted as a novel, evidence-based practice. A literature search around clinicians’ attitudes toward evidence-based practice provides some insights, including that clinicians rate “other” sources of information (eg, colleague opinion and previous experience) as more impactful than published evidence on their decisions for treatment [87-89]. Fostering attitudes conducive to the uptake of evidence-based practice has been associated with transformational leadership styles [80] and systematic training initiatives that highlight how the evidence-based practice is integrated with the wider service system [82], both of which were evident within the current review. However, where there is a disconnect between clinicians and service management or staff does not understand the relative advantage (from the diffusion of innovations) [90], iCBT over existing practice can subsequently create barriers to evidence-based practice uptake [90,91]. This disconnect is well documented in implementation science theories, such as readiness for change [92] and implementation climate [93,94], both of which also emphasize the role of attitudes in evidence-based practice use and implementation.

iCBTs vary widely in their delivery (guided and self-guided), support time frames, and those who provide the support, but the take-home insight is that patients receive the interventions well in terms of satisfaction and clinical outcomes achieved. This malleability of iCBT, where it can assume many forms yet achieve the intended results, underlines the scalability of the intervention. A narrative review of factors associated with scaling public health interventions described that, once an intervention has proven its effectiveness in both small- and large-scale trials, management and practice factors, such as having systems for monitoring intervention performance, funding, and interacting with stakeholders within the wider health care system, become important for the scaling process [95]. Despite this, an “implementation gap” remains, where effectiveness reductions and high levels of attrition occur when we transition from efficacy settings to real-world service provision [96]. This further creates a treatment gap within users assigned iCBT as a treatment option, where implementation factors (eg, abandonment due to encountering bugs and not having a care provider to explore these with) may cause attrition, lowering the promise of scalability purported by these treatments [97].

In total, 4 main limitations were identified as part of this mixed methods systematic review. First, we used a targeted search strategy to produce a dataset that the authors acknowledge is incomplete due to a lack of proper use of terminology within the field to reference implementation. Therefore, we acknowledge that this review is not definitive on the implementation of iCBT and only reports on relevant factors within the articles identified. There is already a movement to standardize the reporting of digitally delivered psychological treatments in research studies (eg, use of CONSORT [Consolidated Standards of Reporting Trials] for eHealth) [98], and perhaps this should be succeeded by an attempt to standardize how we report implementation learnings too.

The second limitation consisted of the “blind spots” associated with the development of the analytic framework that may have resulted from the background of the researchers. The authors mainly come from a background in psychology, and none would consider themselves to be implementation specialists. Other review types (eg, realist, scoping, or narrative reviews) conducted by different research groups may uncover nuances that were otherwise unidentified by this review.

The third limitation of the study relates to the limitations present in the original papers included in this review. The heterogeneous nature of the articles included prevented a formal quality appraisal from being conducted. For instance, we had considered using a tool, such as the Critical Appraisal Skills Programme checklist, but this tool did not provide for a quality review of the narrative-type articles (eg, Schröder et al [56], Andersson et al [57], Andersson et al [58], and others) that were included. Furthermore, there are issues around assessing the quality of qualitative evidence within review-type studies and whether studies should be excluded based on perceived quality, which further compounds the issue [99-101]. Thus, we did not assess the quality of the included articles, and no articles were excluded based on methodological flaws. Relatedly, a few articles had a primary objective of exploring a facet of the implementation of iCBT for depression and anxiety, which is important to note when interpreting the results.

The fourth and final limitation of the study is whether an expansion of search terms could have been used. For instance, the term “computerized CBT” was not used in the search terms. However, while the search string did not include this term, we did include an abbreviated term for this, “ccbt,” and it is notable that by using this abbreviated term we identified specific studies that referenced computerized cognitive behavioral therapy (eg, Wells et al [44], Wright et al [4], Kenicer et al [49], Wright et al [51], and Grist and Cavanagh [54]). Regardless, it is possible that the search string may not have completely captured all possible studies.

This mixed methods systematic review has identified several strategies for consideration when attempting to implement iCBT. Broadly, these strategies emphasize the importance of effective leadership, managing staff and operations associated with the practice of iCBT, implementing and developing professionals to provide the supported component of iCBT, accounting for context, and deriving implementation insights from novel research contributions in iCBT for patient benefit. Future research into iCBT in real-world settings should endeavor to supply appropriate supplemental information that details the efforts associated with implementing the intervention within care pathways. In tandem, efforts could be made to standardize practices that can support the transferability of learning and scalability through the use of a standardized lexicon of terms that are appropriately used.