We developed a comprehensive search strategy for MEDLINE (Ovid; see Multimedia Appendix 1 for MEDLINE [Ovid] search strategy), Embase (Elsevier), Cochrane Central Register of Controlled Trials (CENTRAL) (Wiley), PsycINFO (Ovid), Educational Research Information Centre (Ovid), Cumulative Index to Nursing and Allied Health Literature (Ebsco), and Web of Science Core Collection (Thomson Reuters). Databases were searched from January 1990 to August 2017. We selected 1990 as the starting year for our search because prior to this year, the use of computers was limited to very basic tasks. There were no language restrictions. We searched the reference lists of all the studies that we deemed eligible for inclusion in our review and the relevant systematic reviews. We also searched the International Clinical Trials Registry Platform Search Portal and metaRegister of Controlled Trials to identify unpublished trials.

We developed a common, comprehensive search strategy for a series of systematic reviews focusing on different types of digital education (ie, offline digital education, online digital education, and mobile learning) for preregistration as well as postregistration health care professionals. We retrieved 30,532 records from different bibliographic databases initially. In this review, we only included studies focusing on the effectiveness of offline digital education in medical students’ education, and the findings on other types of digital education (such as virtual reality and mobile learning) within health professions education were reported separately [20-26].

For the purpose of this review, offline digital education can be defined as offline and stand-alone computer-based or computer-assisted learning where internet or intranet connection is not required for the learning activities. Traditional learning can be defined as learning via traditional forms of education such as paper- or text book–based learning and didactic or face-to-face-lecture. Blended digital education can be defined as any intervention that involves the combined use of offline digital education and traditional learning.

We included RCTs and cluster RCTs (cRCTs). Crossover trials were excluded due to a high likelihood of carry-over effect. We included studies with medical students enrolled in a preregistration, university degree program. Participants were not excluded based on age, gender, or any other sociodemographic characteristic.

We included studies in which offline digital education was used to deliver the learning content of the course. This included studies focused solely on offline digital education, or where offline digital education was part of a complex, multicomponent intervention. The main tasks of the learning activities were performed on a personal computer or laptop (with a hard keyboard). The delivery channel of the computer-based intervention was typically accessed via software programs, CD-ROM, DVD, hard disc, or USB memory stick. The focus was mainly on the learning activities that do not have to rely on any internet or online connection. Interventions where the internet connection was essential to provide learning content were excluded from this review.

We included the control groups that comprised traditional learning or traditional face-to-face learning such as lectures or discussions or text- or textbook-based learning as well as other offline digital education. We included studies that compared offline digital education or blended learning to traditional learning or a different form of offline digital education such as CD-ROM or PowerPoint presentations.

Learning outcomes were chosen based on the literature and relevance for medical students’ education [27]. Eligible studies had to report at least one of the specified primary or secondary outcomes. Primary outcomes (measured using any validated or nonvalidated instruments) were medical students’ knowledge scores (postintervention), medical students’ cognitive skills (postintervention), medical students’ postintervention attitudes toward the interventions or new clinical knowledge, and medical students’ postintervention satisfaction with the interventions. Secondary outcomes included the economic impact of offline digital education (eg, cost-effectiveness, implementation cost, and return on investment), changes in its accessibility or availability, and any resulting adverse effects.

The search results from different electronic databases were combined in a single Endnote (X.8.2) library, and duplicate records were removed [28]. Four review authors (BK, GD, MS, and UD) independently screened titles and abstracts of all the records to identify potentially eligible studies. We retrieved full-text copies of the articles deemed potentially relevant. Finally, two reviewers (BK and GD) independently assessed the full-text versions of the retrieved articles against the eligibility criteria. Any disagreements were resolved through discussion between the two reviewers, with a third review author (PP) acting as an arbiter, when necessary.

Five reviewers (BK, GD, MS, UD, and VH) independently extracted relevant characteristics related to participants, intervention, comparators, outcome measures, and results from all the included studies using a standard data-collection form. Any disagreements between the reviewers were resolved by discussion. We contacted the study authors for any missing information, particularly information required to judge the risk of bias.

Four reviewers (BK, GD, MS, and UD) independently assessed the methodological risk of bias of included studies using the Cochrane methodology [29]. The following individual risk-of-bias domains were assessed in the included RCTs: random sequence generation, allocation concealment, blinding (outcome assessment), completeness of outcome data (attrition bias), selective outcome reporting (relevant outcomes reported), and other sources of bias (baseline imbalances).

For cRCTs, we assessed the risk of the following additional domains: recruitment bias, baseline imbalance, loss of clusters, incorrect analysis, and comparability with individually randomized trials recommended by Puffer et al [30]. Judgements concerning the risk of bias for each study were scored as high, low, or unclear. We incorporated the results of the risk-of-bias assessment into the review using a graph and a narrative summary. We also assessed publication bias using a funnel plot for comparisons with at least 10 studies.

For continuous outcomes, we reported mean postintervention scores and SD in each intervention group along with the number of participants and P values. We reported mean postintervention outcome data to ensure consistency across the included studies, as this was the most commonly reported form of findings. We presented outcomes using postintervention standardized mean difference (SMD) and interpreted the effect size using the Cohen rule of thumb (ie, with 0.2 representing a small effect, 0.5 representing a moderate effect, and 0.8 representing a large effect) [29,31]. For dichotomous outcomes, we calculated the risk ratio and 95% CIs. If studies had multiple arms, we compared the most active intervention arm to the least active control arm and assessed the difference in postintervention outcomes. We used the standard method recommended by Higgins et al to convert the results [29].

For meta-analysis, we used a random-effects model. For studies with the same continuous outcome measures, SMDs (for different scales) between groups, along with the 95% CIs, were estimated using Review Manager 5.3 [32]. In the analysis of continuous outcomes and cRCTs, we used the inverse variance method. We displayed the results of the meta-analyses in forest plots that provided effect estimates and 95% CIs for each individual study as well as a pooled effect estimate and 95% CI. For every step in the data analysis, we adhered to the statistical guidelines described by Higgins et al in 2011 [29].

We synthesized the findings from the included studies by the type of comparison: offline digital education (including PowerPoint and CD-ROM) versus traditional learning, offline digital education versus a different form of offline digital education, and blended learning versus traditional learning. Two authors (BK and GD) used the Grading of Recommendations, Assessment, Development, and Evaluations criteria to assess the quality of the evidence [33]. We considered the following criteria to evaluate the quality of the evidence, downgrading the quality where appropriate: limitations of studies (risk of bias), inconsistency of results (heterogeneity), indirectness of the evidence, imprecision (sample size and effect estimate), and publication bias. We prepared a summary of findings table [33] to present the results (Table 1). Where a meta-analysis was unfeasible, we presented the results in a narrative format, such as that used by Chan et al [34].

Knowledge was assessed as the primary outcome in 32 studies [10,36-38,40-67], and majority of the studies (69%) used multiple-choice questions or questionnaires to measure the outcome. Twenty-nine studies assessed knowledge using nonvalidated instruments [36-38,40-49,51-57,59-67], while three studies [10,50,58] used validated instruments such as multiple-choice questions, questionnaires, and tests. Twenty-six studies assessed postintervention knowledge scores, while six studies assessed both short-term and long-term knowledge retention, ranging from 1 week to 22 months of follow-up [52,60,62,65-67] (Multimedia Appendix 3).

Meta-analysis of 19 studies showed that there was no significant difference between offline digital education and traditional learning in postintervention knowledge scores (SMD 0.11, 95% CI –0.11 to 0.32; 1717 participants; small effect size, low-quality evidence; Figure 3). There was a substantial amount of heterogeneity in the pooled analyses (I²=73%). The remaining seven studies were not pooled due to incomplete or incomparable outcome data [40,45,48,49,53,61,65] and reported mixed findings. Four studies reported a significant difference in postintervention knowledge scores in favor of offline digital education (331 participants) [45,48,49,53]. Two studies reported no significant difference between the interventions (289 participants) [40,61], and one study (69 participants) reported a significant difference in postintervention knowledge scores in favor of traditional learning [65]. Taken together, these findings suggest that offline digital education had similar effects as traditional learning on medical students’ postintervention knowledge scores.

One study (26 participants) assessed postintervention knowledge scores in blended learning (offline digital education plus traditional learning) versus traditional learning alone and reported a significant difference in favor of blended learning (SMD 0.81, 95% CI 0.01-1.62; large effect size) [44].

Five studies (478 participants) compared one form of offline digital education (eg, CD-ROM and software programs) to another form of offline digital education (eg, CD-ROM, software programs, or computer-assisted learning programs) [10,43,46,58,66].

Devitt et al reported higher knowledge scores in the computer-based group than a free-text entry program (SMD 1.62, 95% CI 0.93-2.31; large effect size) at 2 weeks postintervention [43]. Taveira-Gomes et al also reported higher postintervention knowledge scores in the computer-based software group (ie, the use of flashcards-based learning materials on cellular structure) compared to a computer-based method alone (ie, without the use of flashcards; SMD 2.17, 95% CI 1.67-2.67, large effect size) [66]. Pusic et al reported that the effectiveness of a simple linear computer program was equivalent to that of a more interactive, branched version of the program in terms of postintervention knowledge scores (SMD 0, 95% CI –0.33 to 0.33; small effect size) [58]. The effect of two studies was uncertain due to incomplete outcome data [10,46]. Overall, the findings were mixed and inconclusive.

Six studies from five reports (605 participants) assessed skills as a primary outcome [35,39,48,65,68]. Three studies used validated instruments to measure the outcome such as an Objective Structured Clinical Examination [68] and a checklist rating form [65]. The remaining three studies used nonvalidated instruments such as questionnaires [35], a six-point checklist scale [39], and self-assessment [48]. Four studies [35,39,48,68] assessed postintervention skills scores, while two studies [65,68] assessed both short-term and long-term skill retention, ranging from 1 month to 10 months of follow-up (Multimedia Appendix 3).

Meta-analysis of four studies showed that, compared with traditional learning, offline digital education improved medical students’ postintervention skill scores (SMD 1.05, 95% CI 0.15-1.95; I² =91%; large effect size; low-quality evidence; Figure 4). There was, however, a considerable amount of heterogeneity in the pooled analyses (I² =91%).

The results of two studies were not pooled due to incomplete outcome data [48,65]. These studies also reported higher postintervention skill scores with offline digital education than with traditional learning [48,65]. Taken together, these results suggest that offline digital education may improve postintervention skill scores compared to traditional learning.