A systematic review was conducted to extract recently published scientific papers that dealt with e-learning in psychotherapy training. We focused on the last 15 years (from 2008 onward) since rapid software and hardware development has occurred during that period although internet delivered and computerized e-learning as a concept emerged as early as the 1990s [25]. This review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [26].

The search was conducted within Ovid MEDLINE, PsycINFO, and Scopus databases. The search strategy included a process where overlapping labels depicting psychotherapy and e-learning were combined. The search was formulated to cover both the training of larger treatment modalities and the specific therapeutic competences. Thus, the search strings were as follows: (psychotherap* or psycholog* or therap* or counsel*) adjacent to (skill* or train* or educat* or compentenc*) OR (interpersonal) adjacent to (skill* or train* or educat* or compentenc*) OR (psychotherap* or psychosoc* or mental health or behavioral health or behavioural health) adjancent to (intervent* or treat* or therap* or evidence-based intervent* or evidence-based treat* or evidence-based therapy* AND (digital* or mobile or web-based or online or distance) adjancent to (learn* or train* or educat* or program* or self-administered learn* or self-administered train* or self-administered educat* or self-administered program*) OR (e-learn* or elearn* or e-educat*). The searches were conducted on all databases in June 2022. To ensure optimal coverage considering the field’s unestablished terminology, a comprehensive array of search terms was used.

Examination of the titles and abstracts of the publications and the assessment for potential inclusion were carried out by first (KM) and last author (SES). Data extraction was carried out by the first author and included (1) target group of the training; (2) description of training content; (3) description of e-learning methods; (4) comparison between training modalities; (5) sample sizes; (6) primary outcome; and (7) summary of results. Furthermore, to calculate a measure of effect size for comparing differences between conventional training methods and e-learning, the means and standard deviations of the learning outcomes reported in the studies were extracted.

The risk of bias was assessed with the latest Cochrane risk-of-bias tools: RoB 2—tool for randomized trials and ROBINS-I (Risk of Bias in Non-randomized Studies of Interventions) for nonrandomized trials [27,28]. The risk of bias analysis was systematically conducted by the first (KM) and last (SES) authors in an independent manner. Subsequent to this individual assessment, any emergent discrepancies were collaboratively addressed through discussions focused on achieving consensus.

We classified the results of the primary outcomes of the included studies using the Kirkpatrick model of training evaluation [29], which evaluates training outcomes on 4 levels. The first level (“reaction”) evaluates the training participants’ immediate reaction and feedback regarding the training program including satisfaction, engagement, and relevance. The second (“learning”) evaluates the learning outcomes of the training program including change in attitudes, confidence, knowledge, and skills. The third level (“behavior”) evaluates how much the trainee participants apply the learned knowledge and skills when treating patients in real life. The final level (“results”) evaluates how much the patient outcomes improve because of the training.

In order to quantify the magnitude of the difference observed in these results, Hedges g was calculated as a measure of effect size with 95% CIs [30]. Hedges g is comparable with Cohen d but corrects small sample bias. Therefore, Hedges g of 0.2 represents a small effect; 0.5, a moderate effect; and 0.8, a large effect [31]. The effect size was calculated from the studies that included the figures necessary for the calculations. For studies with missing information, the first author of the publication was contacted, and they were asked for the data. An overall effect size describing the equivalence between training methods was formulated as an arithmetic mean of all calculated effect sizes.

The first screening from the databases identified 3380 articles. Duplicates (n=697) were removed using the “Merge items” tool in the “Duplicate items”—folder in “Zotero,” which the authors used as a reference management software. After removing the duplicates, 2683 articles were identified and the eligibility was assessed by KM based on their title, and abstract if necessary, and 2530 articles were excluded as clearly fulfilling the exclusion criteria mentioned in the “Selection Criteria” section of the article (see Figure 1). The remaining 153 articles were read in full-text versions by KM and 79 articles were excluded according to the same criteria. KM reviewed the remaining 74 articles in full, and SES reviewed in full, a random sample of those articles (n=45). The size of random sample required was calculated using Cohen κ for interrater agreement level with the assumptions that 30%-40% of the remaining articles would be included in the final sample, and the interrater agreement is moderate at the least [32]. The calculations were conducted with R with the “CIBinary”—function within the “kappaSize”—library. Based on these calculations, the last author reviewed the random sample of 45 articles. The authors worked blind to each other. The results were compared, and the authors were in substantial agreement on which articles to include and exclude (Cohen κ=0.73). All discrepancies were discussed to a consensus (eg, a detailed discussion whether the target of a training described in a specific study was related to psychotherapy skills or more general interaction skills used in health care). Thirty-four studies were included in the final review.

The articles included in the study are listed in Table 1 for randomized controlled trials (n=17) and Table 2 for nonrandomized (n=17) studies. The tables include the target group of the training, the content of the training, a depiction of the e-learning system used, a comparison between training modalities if applicable, the primary outcomes of the studies, and a summary of the main findings. In all studies, training was targeted to health care providers. The training content was most often related to some form of cognitive behavioral therapy (CBT) and the length of the e-learning programs ranged from 15 minutes to 52 hours. In the studies included, the effects of e-learning are evaluated in within-trainee designs and comparisons with waiting for training and other active training methods. The data were extracted by the first author as they were presented in the original studies.

Risk of bias was assessed both for the randomized and nonrandomized studies included in the review using the Cochrane risk of bias tools (RoB 2 and ROBINS-I). The risk of bias analysis was carried out by the first and last authors (KM, SES). The authors agreed upon the used tools and familiarized themselves with the logic behind them. The first author did the risk of bias analysis and classification. Then the last author examined the classification to identify potential discrepancies. No discrepancies occurred. In the review, 13 out of the 17 randomized studies, which corresponds to 76%, have been categorized as exhibiting a low risk of bias. One of the studies (6%) showed some concerns regarding the risk of bias. Three of the studies (17%) were flagged for having a high risk of bias. One of these studies had a source of bias from the randomization process and the 2 others have a potential source of bias from a large amount of missing outcome data. These results can be viewed in Figure 2. The ROBINS-I classification for risk of bias in nonrandomized studies included in the review can be viewed in Multimedia Appendix 1.

Among the nonrandomized studies included in the review, 16 out of 17 (94%) were flagged for having either a serious or critical risk of bias. The source of bias in all these studies was, at the very least, inadequate control of potential confounding variables. One study [58] was flagged at being only at moderate risk of bias.

Thirteen out of 34 (38%) studies, in which 9 were randomized trials [34,35,39-42,46,48,49] and 4 were nonrandomized [52,55,58,63], reported a comparison between a web-based training and training carried out by conventional methods. Of the 13 studies examined, 2 (15%) [42,48] reported results suggesting superiority (related to knowledge acquisition outcomes) for e-learning compared with conventional training. Meanwhile, 3 (23%) [40,42,52] indicated inferiority (concerning outcomes related to skills acquisition, satisfaction, self-efficacy, motivation, opportunity for active participation, and trainee-assessed professional benefit). In addition, 12 (92%) [34,35,39,41,42,46,48,49,52,55,58,63] found no significant difference between e-learning and conventional training methods, reporting equivalence (in terms of outcomes related to knowledge acquisition, skills acquisition, clinical application of the methods trained, self-efficacy, satisfaction, and acceptance rates) at posttraining evaluations. It is worth noting that studies reporting multiple outcomes could be included in more than 1 category.

Eight out of these 13 studies included the necessary figures to calculate Hedges g. The first authors of the remaining studies were contacted, but none provided the figures or raw data within an acceptable time window of 1 month. The overall mean effect size from all comparisons within the 8 studies (n=30) was 0.01. A forest plot displaying effect sizes and their corresponding 95% CIs is shown in Figure 3. The results are organized and presented in accordance with the Kirkpatrick model categorization.

A systematic literature review of e-learning in the context of psychotherapy training was conducted following the PRISMA guidelines. E-Learning was positively associated with learning outcomes in terms of knowledge and skills acquisition as well as implementing learned skills to practice according to the Kirkpatrick levels 1-3. There were no apparent differences between e-learning and conventional training methods. None of the studies examined the impact of training on patient outcomes, and the research demonstrated significant heterogeneity.

This is the first study to exclusively assess the level of research regarding e-learning within this domain. The review included 34 studies conducted between 2008 and 2022, encompassing a mix of both randomized and nonrandomized research. Up to this point, findings on training outcomes have been reported across various outcomes and the level of evidence is still emerging. The target group, training content, a description of e-learning, sample sizes, primary outcomes, and a summary of results were provided from each article included in the review. Most of the randomized studies were linked with a low risk of bias, whereas nonrandomized studies exhibited a severe risk of bias. The risk of bias in the majority of the nonrandomized studies stemmed from limited control of confounding variables. The execution method of e-learning, the measurement methods used in training outcome assessments, and the trainees were highly heterogenous across the included studies. The high amount of heterogeneity in various aspects of the included literature and small sample sizes in many of the included studies create challenges in synthesizing the findings and drawing clear conclusions. Therefore, it is necessary to process the literature using a higher-lever categorization in learning outcomes (such as the Kirkpatrick model).

e-Learning within psychotherapy training showed positive associations with several aspects: trainee satisfaction, enhanced learning outcomes—including knowledge and skills acquisition, and the application of trained methods into practice, as observed in within-trainee designs. These findings align with levels 1-3 of the Kirkpatrick model. No outcomes related to patient outcomes were reported (Kirkpatrick level 4). The results of this review point in the direction of equivalence between e-learning and conventional training methods such as lectures and workshops. Training outcomes including trainee satisfaction, knowledge, and skill acquisition, and application of trained content in clinical work were comparable between e-learning and conventional training. The overall effect size signifying this difference was 0.01, indicating no difference.

In the only study that reported superiority of conventional training methods in skills acquisition e-learning involved solely watching previously recorded lectures rather than actively interacting with content specifically designed as an e-learning program [40]. In the other studies, where the inferiority of e-learning was presented as a result with a more modern approach on e-learning, the inferiority was in the context of trainee satisfaction, self-efficacy, and motivation [42,52]. E-Learning was reported to be superior compared with conventional training only in the context of knowledge acquisition when training DBT [42,48].

The trainees and the clinical contexts of the trainees varied heavily between included studies. Furthermore, the operationalization of the learning outcomes and the associated measurement methods were not consistent and included both subjective self-ratings of knowledge and competence and more objective measures (eg, quizzes and expert-rated competence scores). In most of the studies, the training content included CBT-based methods, and in some studies interpersonal therapy and short-term psychodynamic therapy. Most of the e-learning solutions described were similar and included case vignettes, demonstrations, multimedia didactic content, and various assignments.

This review has many strengths. These include following the PRISMA guidelines, giving an exhaustive look in the current literature, and that the results of research comparing e-learning and conventional training methods were uniformly aggregated behind a single statistic. As anticipated, studies identified in this review had high heterogeneity of learning methods, therapy content, and outcomes, making systematic comparison difficult. We tried to control this heterogeneity by using the Kirkpatrick model of training evaluation, summarizing the results by calculating effect sizes, and requesting original data from authors to do that. However, it seems that heterogeneity is a feature of this field of study and cannot be easily overcome with post hoc methods; a more uniform research methodology would be needed in future studies.

This systematic review also has limitations. First, the review did not include gray literature (materials and research produced outside of academic publishing) or literature beyond the scope of the conducted search, which may include information relevant to this study. Second, in addition to the risk of bias assessment, the quality of the research included was not systematically assessed (eg, publication bias). Third, the search method yielded a substantial amount of literature not relevant to the study, which makes the replication of this review challenging. This is mainly attributed to the overlap in vocabulary between literature related to training and psychosocial interventions themselves. This issue also touches upon a possible methodological limitation in our screening process, namely, that the initial screening phases were conducted by a sole reviewer. The purpose of this preliminary screening was to promptly identify and eliminate studies that were unmistakably irrelevant to our review, with a large proportion of such studies investigating digital psychosocial interventions (eg, internet-based cognitive behavioral therapy), a testament to the prevalent overlap in terminologies used. To ensure a balanced review, the authors applied a statistical approach to select a representative sample of studies for evaluation by the second author. We posit that the inclusion of 2 independent raters from the outset, as opposed to 1, during the initial screening phases might have had a negligible effect on the outcomes, considering that all studies with any element of ambiguity were carried forward for full review. Nonetheless, we acknowledge that our approach may be more susceptible to bias compared with methodologies that use multiple independent reviewers for the assessment of all studies. We consider this as a potential limitation and agree that the use of dual raters throughout the process might augment the robustness against bias.

Based on this literature review, we conclude that positive learning outcomes are generally associated with various e-learning programs in psychotherapy training including trainee satisfaction, knowledge and skill acquisition, and in application of trained content in clinical practice. Learning outcomes are generally equivalent between e-learning and conventional training methods. This equivalence is interpreted as a supportive finding for e-learning, considering the other benefits of e-learning, such as scalability and consistency of quality. While more research is warranted, and research methodology needs to be more consistent in the description of e-learning solutions and in learning outcome assessment, it seems advisable to use e-learning as a part of psychotherapy training curriculums.

e-Learning creates an opportunity to scale up training in large and even in nationwide settings. This, in turn, can significantly improve equal and sufficient access to psychotherapy in public health care settings, which are impossible with conventional and unscalable training methods [61]. This can be seen, for example, within the study by Wilkerson et al [50], where it was estimated that 2500 individuals will annually complete an e-learning program in Cognitive Behavioral Therapy for Insomnia, a scale the authors declared as “massive dissemination.” With centralized digital solutions, e-learning might prove itself as the more cost-effective solution than conventional training methods while ensuring quality in training as many learning outcomes are comparable between these methods.

As the literature is still emerging and current corpus is displaying a high amount of heterogeneity, small sample sizes, and inconsistent description of e-learning solutions and measurement methods in learning outcomes, it would be advisable to use e-learning as a part of blended training programs. By comparison with e-learning results in health education in general, a blended training model might even be the most optimal way to implement e-learning. E-Learning is generally recognized as an opportunity to improve the availability of training programs by eliminating constraints set by time and geographical location and also a way to increase the quality of training programs by shifting the focus of the surrounding pedagogics toward more learning-centric direction. Knowledge acquisition can be supported through e-learning, but refining skills and ensuring competencies may require synchronous interaction with a trainer to reach optimal learning outcomes [67].

In psychotherapy training, a shift toward minimizing the didactic approach of trainers by facilitating the active role of the trainee in knowledge acquisition is warranted. E-Learning viewed as a pedagogy rather than technology may be an optimal solution in implementing flipped classroom and other activating learning methods. Within these methods, synchronous learning events are held with trainees who have already familiarized themselves with the necessary materials and focus on honing therapeutic and reflective skills and addressing student motivation. Future research needs to focus on replicating the effects of e-learning in trainee satisfaction, learning and behavior, and on studying the effectiveness and cost-effectiveness of training (e-learning or conventional methods) on patient outcomes. Research is especially needed to further determine the optimal way to combine e-learning and conventional training methods. Furthermore, to reach the full potential of e-learning in the context of psychotherapy training, international collaboration is warranted to bring consistency on how e-learning systems are developed, described, and assessed.