A search strategy was developed in accordance with the Cochrane Handbook of Systematic Reviews of Interventions [39] to search the Cochrane CENTRAL, MEDLINE (Ovid), Embase (Elsevier), PsycINFO (Ovid), ERIC (Ovid), CINAHL (Ebsco), Web of Science Core Collection (Thomson Reuters), and International Clinical Trials Platform (World Health Organization) databases. The detailed search strategy is presented in Multimedia Appendix 1, and a detailed description of the methodology has been published elsewhere [40]. Databases were searched from January 1, 1990, to March 9, 2017. We selected 1990 as the starting year for our search because prior to this, the use of ICT for education was limited. We identified additional studies both by scanning relevant systematic reviews and meta-analyses and hand searching reference lists of all included studies.

Only randomized controlled trials (RCTs), cluster RCTs (cRCTs), and quasi-randomized controlled trials of postregistration education for medical doctors using ODE (standalone or blended) with any type of controls measuring knowledge, cognitive skills, attitudes, satisfaction (primary outcome), changes in practice or behavior, patient outcomes, costs, or adverse effects (secondary outcome) outcomes were eligible for inclusion in this review (Multimedia Appendix 2). Participants were not excluded on the basis of age, gender, or any other sociodemographic variables. We used the gold-standard systematic review methods recommended by the Cochrane Collaboration and strictly adhered to the published protocol [38].

Three reviewers (PG, OZ, and BK) independently screened the titles and abstracts and full-text versions of the eligible studies and performed the data extraction. We extracted the relevant data on participants, intervention and control, outcome measures, and study designs. We contacted the study authors in cases of any missing information. A fourth review author (PP) acted as an arbiter in cases of disagreement.

Two reviewers independently assessed the risk of bias for RCTs using the Cochrane Collaboration’s “risk of bias” tool [39]. For RCTs, we did so across the domains of random sequence generation, allocation concealment, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, and other bias including the comparability of intervention and control group; characteristics at baseline; validity and reliability of outcome assessment tools; and protection against contamination. Blinding of participants and personnel was not assessed, as the nature of the intervention precludes blinding.

We assessed the risk of bias for cRCTs across the domains of recruitment bias [41], baseline imbalances, loss of clusters incorrect analysis, and comparability with individual randomized trials [39]. For each study, two reviewers independently categorized each domain as low, high, or unclear risk of bias.

Clinical heterogeneity was assessed to check if the included studies were similar in terms of their population, intervention characteristics, and reported outcomes and to ascertain the possibility of pooling the measures of effect. The extracted data were analyzed using RevMan 5.3 software (The Nordic Cochrane Centre, Copenhagen, Denmark). Statistical heterogeneity was assessed using the Chi-square and I² tests [39]. We found significant heterogeneity (clinical and statistical) among the included studies; hence, meta-analysis was not suitable for analysis.

The results from individual RCTs were reported as the standardized mean difference (SMD) for continuous variables and risk ratios (RR) for dichotomous variables. Where studies reported more than one measure for each outcome, the primary measure, as defined by the study authors, was used in the analysis. If studies had multiple arms, we compared the intervention arm to the least active control arm and assessed the difference in postintervention outcomes. Similarly, when multiple domains of the same outcome were measured, only the primary domains identified and agreed upon by the review authors were reported. Meta-analyses were not possible because there was significant clinical and methodological heterogeneity across the included studies.

Summary of findings tables (Tables 1-3) were prepared based on the methods described in Chapter 11 of the Cochrane Handbook for Systematic Reviews of Interventions [39]. Two review authors (PG and BK) independently used the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) criteria to rank the quality of the evidence using the GRADE profiler (GRADEpro) software [42]. In the main review, we only compared ODE with self-directed/face-to-face learning and blended learning with self-directed/face-to-face learning; the rest of the comparisons are presented in Multimedia Appendix 3.

Our searches yielded a total of 27,488 citations and 93 studies (Figure 1). Of those, 74 studies were RCTs including 12,537 participants and 19 were cRCTs including 1262 clusters, 3727 physicians, and 7690 patients. Sixty-four studies were published between 2010 and 2017, and the remaining 29 studies were published between 1999 and 2009.

A total of 29 (31.1%) studies were conducted among primary care practitioners (general practitioners, family medicine practitioners/residents, and occupational physicians), 12 among surgeons (12.9%), 11 among general and internal medicine practitioners (11.8%), and 8 among pediatricians (8.6%; Figure 2 and Multimedia Appendices 4-10). Only 2 (2.2%) [83,96] of the 93 studies were conducted in low- to middle-income countries; all the remaining studies were conducted in high-income countries with majority in the United States (53.8%), Canada (10.8%), and Germany (5.4%; Multimedia Appendix 11). Fifty studies were carried out in hospital settings, 31 studies were conducted in university settings, 11 studies were conducted in primary care settings, and one study was conducted in a mixed hospital and university setting.

A total of 61 studies compared ODE and self-directed/face-to-face learning. Self-directed learning was defined as self-learning through books and journals, and face-to-face learning was defined as learning through didactic classroom lectures and courses. Fourteen studies compared ODE and other types of ODE, 3 studies compared ODE and blended learning, and 15 studies compared blended learning and self-directed/face-to-face learning. Two studies used synchronous learning technology (video-conferencing systems) for training, and 39 studies used asynchronous learning technologies such as Web-based libraries/repositories of video modules, CD-ROM, emails, and online discussion groups to deliver the intervention. In the main review, we only compared ODE with self-directed/face-to-face learning and blended learning with self-directed/face-to-face learning; the rest of the comparisons are presented in Multimedia Appendix 3.

A total of 51 studies were considered to be at high risk of bias for at least one of the risk of bias domains (Figures 3 and 4). Six studies were rated as having a high risk of selection bias, 31 studies were rated as having a high risk of attrition bias due to a high drop-out rate (>20%), and 3 studies [103,107,108] had a high risk of reporting bias. Epstein et al reported a high risk of detection bias [109]. Further, 25 studies had a high risk of “other biases.” Similarly, among the cRCTs, 12 studies had a high risk of bias for baseline imbalance, 8 studies had a high risk of bias for loss of clusters, and 3 studies had a high risk of bias for incorrect analyses. Risk of bias is described in detail in Multimedia Appendices 4 and 12.

The characteristics of included studies categorized by participants’ specialty, outcomes, comparisons, and intervention types are presented in Multimedia Appendices 5-13. The educational content was heterogeneous among the included studies. Studies that compared ODE/blended ODE with self-directed or face-to-face learning are presented in the manuscript; for other comparisons, see Multimedia Appendix 3.

The review identified a variety of ODE interventions used for postregistration training of medical doctors; these interventions were used among diverse medical specialties, and a range of outcomes and comparators were reported. Because of the high heterogeneity, we were unable to pool the data quantitatively.

A total of 93 RCTs were included, of which 76 studies compared ODE/blended learning with self-directed/face-to-face learning (N=12,424). The results are presented in the main review, and the rest of the comparisons are presented in the Multimedia Appendices 1-13. Among these, 21 studies with 2611 participants reported higher postintervention knowledge scores (small to large effect size, very low quality) for the intervention group, while 20 studies with 5496 participants reported no difference in knowledge scores between the groups. Seven studies with 794 participants reported higher postintervention skill scores in the intervention group (large effect size, low quality), while 13 studies with 4447 participants reported no difference in skill scores between the groups. One study with 99 participants reported higher postintervention attitude scores with the intervention (very low quality), while 4 studies with 305 participants reported no difference in attitude scores between the groups. Four studies with 677 participants reported higher postintervention physician satisfaction with the intervention (large effect size, low quality), while 6 studies with 478 participants reported no difference in satisfaction between the groups. Eight studies with 5051 participants reported higher postintervention practice or behavior change for the ODE group (small to moderate effect size, low quality), while 5 studies with 377 participants reported no difference in practice or behavior change between the groups. One study with 449 participants reported higher improvement in patient outcome, while 3 studies with 667 participants reported no difference in patient outcome for the groups. None of the included studies reported any unintended/adverse effects of the interventions on learners.

Overall, the effect of ODE on postintervention knowledge, skills, attitude, satisfaction, practice or behavior change, and patient outcomes was inconsistent and ranged mostly from no difference between the intervention groups to higher postintervention score in the ODE/blended learning group (small to large effect size, very low to low quality evidence). Moreover, the quality of evidence according to GRADE criteria was judged to be low for most outcomes.

George et al assessed the effectiveness of ODE for undergraduate health professionals [10] and reported that ODE is equivalent, and possibly superior, to traditional learning. We are not aware of any other systematic reviews of RCTs that have evaluated ODE for medical doctors. A similar review of evidence from nonrandomized studies on the effectiveness of ODE in surgical education among medical and dental students, surgeons, and oral health specialists [121] reported knowledge gain from ODE compared with active control (face-to-face learning) or no intervention (self-directed learning). Another review [122] evaluated the evidence of effectiveness from nonrandomized studies of online CME for general practitioners alone; assessed their satisfaction, knowledge, clinical practice, and patient outcomes; and reported improvement in satisfaction, knowledge, or practices. Jwayyed et al [123] evaluated the effectiveness of ODE from nonrandomized studies among diverse health care professionals and reported inconsistent results. Our review, in congruence with other reviews [121-124], compared the effect of ODE and blended learning with self-directed/face-to-face learning and other forms of ODEs on physicians’ knowledge, skills, attitudes, satisfaction and clinical practice, and patient outcomes, but only included evidence from RCTs and cRCTs.

According to the GRADE criteria, the quality of the evidence was very low for knowledge and low for the other primary and secondary outcomes due to the unclear and high risk of bias, inconsistency, and publication bias. The majority of the studies did not provide information on randomization sequence generation and allocation concealment. Similarly, a high proportion of studies (75 studies, 82%; including comparisons presented in the Multimedia Appendix 12) did not provide sufficient information on the blinding of outcome assessors and were hence judged to have an unclear risk of bias. Thirty-one studies (33.3%) reported incomplete outcome data, 25 studies (26.9%) reported baseline differences in participant characteristics and were judged to be at high risk of bias, and 3 studies (3.3%) had a high risk of reporting bias.

This review has a few limitations. The included studies were heterogeneous in terms of the participants, learning content, and the types of ODE (CME/CPD), thus limiting the opportunity to pool the results and consequently run the preplanned subgroup analysis. Hence, the review could not generate conclusive findings on the effectiveness of ODE. Of the included studies, only two were from low- to middle-income countries, which could limit the completeness of the evidence and its generalizability in all settings. Third, we were unable to assess the cost-effectiveness of ODE as compared to self-directed/face-to-face learning, because none of the identified studies formally assessed it. Only three studies assessed the cost and maintenance of the ODE intervention. Fourth, none of the studies specifically addressed any adverse effects of ODE.

The study has several strengths including a thorough and reproducible search of available literature, independent screening and data extractions, and critical appraisal of the literature conducted in accordance with the Cochrane Handbook for Systematic Reviews of Interventions.

Given the strength of the evidence of ODE and blended learning, especially for cognitive, procedural, and diagnostic training; evidence-based medicine training; and CME, CPD, and Clinical Practice Guideline training, further high-quality studies of ODE interventions with a validated learning theory are needed to establish their effectiveness, cost-effectiveness, and financial sustainability (return on investment). Specifically, these studies should seek to address several unanswered questions such as ODE’s theoretical underpinning, content validation, frequency, intensity, interactivity, technical features, fidelity, safety, cost-effectiveness, adaptability, acceptability, barriers/facilitators to its adoption, financial sustainability, and learner’s readiness to switch from classroom learning to complete ODE.

Our review found that ODE and blended learning refers to a group of heterogeneous interventions with different learning theories, learning content, comparators, and outcomes. These interventions were used to train medical doctors in various specialties, such as primary care practitioners, surgeons, residents, and physicians. Although empirical evidence from a majority of studies shows that ODE and blended learning may improve practicing physicians’ knowledge, skills, attitude, satisfaction, practice or behavior change, and patient outcomes, few other studies showed that they may be comparable to self-directed, face-to-face learning. The quality of the evidence in these studies was found to be low and very low for knowledge. Therefore, further high-quality RCTs are required before the evidence of efficacy can be concluded for postregistration training of medical doctors.