This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The use of digital education in problem-based learning, or digital problem-based learning (DPBL), is increasingly employed in health professions education. DPBL includes purely digitally delivered as well as blended problem-based learning, wherein digital and face-to-face learning are combined.
The aim of this review is to evaluate the effectiveness of DPBL in improving health professionals’ knowledge, skills, attitudes, and satisfaction.
We used the gold-standard Cochrane methods to conduct a systematic review of randomized controlled trials (RCTs). We included studies that compared the effectiveness of DPBL with traditional learning methods or other forms of digital education in improving health professionals’ knowledge, skills, attitudes, and satisfaction. Two authors independently screened studies, extracted data, and assessed the risk of bias. We contacted study authors for additional information, if necessary. We used the random-effects model in the meta-analyses.
Nine RCTs involving 890 preregistration health professionals were included. Digital technology was mostly employed for presentation of problems. In three studies, PBL was delivered fully online. Digital technology modalities spanned online learning, offline learning, virtual reality, and virtual patients. The control groups consisted of traditional PBL and traditional learning. The pooled analysis of seven studies comparing the effect of DPBL and traditional PBL reported little or no difference in postintervention knowledge outcomes (standardized mean difference [SMD] 0.19, 95% CI 0.00-0.38). The pooled analysis of three studies comparing the effect of DPBL to traditional learning on postintervention knowledge outcomes favored DPBL (SMD 0.67, 95% CI 0.14-1.19). For skill development, the pooled analysis of two studies comparing DPBL to traditional PBL favored DPBL (SMD 0.30, 95% CI 0.07-0.54). Findings on attitudes and satisfaction outcomes were mixed. The included studies mostly had an unclear risk of bias.
Our findings suggest that DPBL is as effective as traditional PBL and more effective than traditional learning in improving knowledge. DPBL may be more effective than traditional learning or traditional PBL in improving skills. Further studies should evaluate the use of digital technology for the delivery of other PBL components as well as PBL overall.
Problem-based learning (PBL) has been used as an educational approach in health professions education in many medical and nursing school curricula worldwide for over 50 years [
Worldwide, various components of PBL are being increasingly delivered using digital technology. Digital education is changing the way in which health professions education, including PBL, is conducted. Digital education may comprise a variety of interventions based on learning tools, theories, content, objectives, teaching methods, and setting of delivery. In terms of the type of learning technologies, digital education includes, but is not restricted to, online and offline computer-based learning, massive open online courses, virtual reality, virtual patient simulation, mobile learning, serious gaming and gamification, and psychomotor skills trainers (
Digital problem-based learning (DPBL), or the use of different types of digital technologies to deliver PBL, has the potential to enhance the authenticity, appeal, accessibility, and effectiveness of PBL by enhancing participants’ communication, collaboration and self-learning [
We followed the Cochrane methodology for every step of the review [
In this review, we included randomized controlled trials (RCTs) that evaluated the effectiveness of DPBL in improving health professionals’ knowledge, skills, attitudes, and satisfaction of students and compared DPBL with traditional learning methods or other forms of digital learning. Crossover trials were excluded because of a high likelihood of a carry-over effect.
We included studies with preregistration as well as postregistration health professionals as per the qualifications listed in the Health Field of Education and Training (091) of the International Standard Classification of Education [
We included studies in which any form of digital technology was used in combination with PBL for delivering the learning content of courses, either as the sole (full digital learning) or partial (blended learning) means of delivery, for the purpose of learning in health professions education. Digital technology primarily supports PBL principles and processes by enabling contextual and collaborative learning [
We analyzed studies that compared DPBL to traditional PBL or traditional learning (textbook, lectures, etc) and to different forms of DPBL interventions or other digital education.
We excluded studies that focused on individual learning interventions, evaluated the use of DPBL in other educational areas, lacked an active comparison, and assessed interventions with optional or minimal use of digital technology.
We included the following primary outcomes: (1) students’ postintervention cognitive knowledge and skills measured with any instrument (validated or nonvalidated); (2) students’ professional postintervention attitudes toward DPBL interventions, patients, or new clinical knowledge or skills measured using any instruments (validated or nonvalidated); students’ postintervention satisfaction with DPBL intervention, measured using any instrument (validated or nonvalidated).
For secondary outcomes, we focused on the economic impact of the DPBL intervention and potential adverse or unintended effect of the DPBL intervention.
We searched seven electronic databases, namely, MEDLINE (Ovid), Embase (Elsevier), Cochrane Central Register of Controlled Trials (CENTRAL, Wiley), PsycINFO (Ovid), Educational Research Information Centre (Ovid), Cumulative Index to Nursing and Allied Health Literature (Ebsco), and Web of Science Core Collection (Thomson Reuters), for relevant studies from January 1990 to August 16, 2017, without language restrictions (
We also checked the reference lists of all included studies and relevant systematic reviews and searched the International Clinical Trials Registry and metaRegister of Controlled Trials for unpublished trials. We followed the Cochrane methodology for the selection of studies, data extraction, data analysis, and risk of bias analysis, with two reviewers independently performing each of these steps [
Standardized mean difference (SMD) for continuous outcomes, with 95% CI, was calculated based on the availability of data from the included studies. We pooled studies together based on comparison and outcomes using SMD. We interpreted the effect size using the Cohen rule of thumb (ie, with <0.2 representing no effect, 0.2 to <0.5 representing a small effect, 0.5 to <0.8 representing a moderate effect, and ≥0.8 representing a large effect) [
The search strategy yielded 30,532 references. We included nine studies with 890 medical students (
The characteristics of the nine included studies are presented in
In most studies, DPBL interventions were delivered face to face and digital technology was employed for one component of the PBL process—presentation of problems. In four studies, DPBL was at least partially delivered on a distance basis. In one study, DPBL was employed for delivery of the initial part of the PBL (ie, the first meeting;
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram. RCT: randomized controlled trial.
The types of digital education modalities included offline learning (eg, CD-ROM) [
Included studies reported findings on students’ knowledge, skills, attitude, and satisfaction. No studies reported cost-related outcomes or adverse/unintended effects of DPBL-based interventions. All studies measured outcomes immediately after the intervention, except Sobocan et al [
The effects of DPBL compared to the traditional PBL on knowledge scores were reported in eight studies involving 822 medical students (
We also performed subgroup analysis based on the degree to which the digital technology was employed as part of PBL. We differentiated among studies in which digital technology was used for presentation of problems [
Characteristics of the included studies.
Comparisons groups and studies | Learning modalities compared | Number and types of participants | Field of study | Outcomes | |
Alverson et al 2008 [ |
VRd PBL vs traditional PBL | 36 medical students (year unspecified) | Traumatic head injury | Knowledge | |
Bowdish et al 2003 [ |
Online PBL vs traditional PBL | 150 medical students (first year) | Human physiology | Knowledge | |
Dennis 2003 [ |
Online PBL vs traditional PBL | 34 medical students (second year) | Pregnancy-associated urinary incontinence | Knowledge | |
Kong et al 2009 [ |
Online PBL vs traditional PBL | 90 medical students (year unspecified) | Ophthalmology | Knowledge | |
Li et al 2013 [ |
Offline PBL vs traditional PBL | 120 medical students (fourth year) | Dermatology | Knowledge, Skills | |
Moeller et al 2010 [ |
Online PBL vs traditional PBL | 237 medical students (year unspecified) | Multidisciplinary | Knowledge, Skills | |
Sobocan et al 2017 [ |
VPe-based PBL vs traditional PBL | 34 medical students (third year) | Internal medicine | Knowledge | |
Taradi et al 2005 [ |
Online PBL vs traditional PBL | 121 medical students (second year) | Biochemistry (acid-base physiology) | Knowledge | |
Kong et al 2009 [ |
Online PBL vs traditional learning (lecture) | 90 medical students (year unspecified) | Ophthalmology | Knowledge | |
Li et al 2013 [ |
Offline PBL vs traditional learning (lecture) | 120 medical students (fourth year) | Dermatology | Knowledge | |
Schutte et al 1997 [ |
VR PBL vs traditional learning (textbook) | 68 medical students (first year) | Genetics (global structure of DNA) | Knowledge |
aDPBL: digital problem-based learning.
bPBL: problem-based learning.
cRCT: randomized controlled trial.
dVR: virtual reality.
eVP: virtual patient.
The effects of DPBL compared to the traditional PBL on skills scores were reported in two studies (N=357). The pooled analysis of these two studies showed that DPBL may slightly improve postintervention skill scores (SMD 0.30, 95% CI 0.07-0.54;
The effects of DPBL compared to the traditional PBL on satisfaction scores were reported in three studies with mixed findings. Two studies evaluating the use of DPBL with digitally presented problems reported no difference between DPBL and traditional PBL in satisfaction scores [
The effects of DPBL compared to traditional learning on knowledge scores were reported in three studies (N=278) [
The effects of DPBL compared to traditional learning on skills scores were reported in one study. This study [
A conceptual framework for the use of digital technology in PBL. PBL: problem-based learning.
Risk-of-bias summary: review authors' judgement about each risk-of-bias item for each included study.
The effect of DPBL compared to traditional PBL (knowledge outcome, postintervention). DPBL: digital problem-based learning; PBL: problem-based learning; IV: interval variables, Random: random effect model.
The effect of DPBL compared to traditional learning (knowledge outcome, postintervention). IV: interval variables; Random: random effect model; DPBL: digital problem-based learning.
In this review, we evaluated the effectiveness of the use of digital technology for delivering PBL. Our findings show that DPBL improves students’ postintervention knowledge scores in comparison to traditional learning. DPBL is as effective as traditional PBL in improving students’ postintervention knowledge and may slightly improve postintervention skills. Moreover, fully digitally delivered, distance-based DPBL may lead to better knowledge scores in comparison to traditional PBL. The risk of bias in the included studies was mostly judged as unclear due to a lack of information on randomization, allocation concealment, and outcome assessment blinding. In the included studies, the term “blended PBL” was employed to denote diverse configurations of digital technology and PBL. For example, an intervention in which PBL was fully delivered online, but included one visit to the clinic, and another intervention in which face-to-face delivered PBL included digital presentation of problems were both termed blended learning. The use of “blended learning” therefore seemed misleading in this context. We decided to focus primarily on describing the way in which digital technology was employed in the PBL process. From our viewpoint, there are two main applications of digital technology in PBL: full or partial delivery of distance-based PBL or support of delivery of different components of face-to-face or colocated PBL. Most studies included in our review focused on colocated PBL, with digital technology used for the presentation of problems. This corresponds to the findings from two reviews focusing on the application of digital technology in PBL [
Our findings show that DPBL was more effective than traditional learning. Although we were unable to find meta-analyses comparing DPBL to other forms of education, there are numerous meta-analysis comparing traditional PBL to traditional learning [
We found that DPBL was as effective as PBL in terms of knowledge, and fully digitally delivered distance-based DPBL was potentially more effective than traditional PBL. Although there are some nonrandomized studies on distance-based PBL corroborating this finding, the evidence from RCTs is scarce [
Our review has several limitations. Although RCTs provide the highest level of evidence for the effectiveness of an intervention, it is not always possible to use a rigorous RCT approach in educational research [
DPBL includes diverse applications of digital technology as part of face-to-face as well as distance-based PBL. Our findings suggest that DPBL is more effective than traditional learning and as effective as traditional PBL in improving postintervention knowledge outcomes. For improvement of skill outcomes, DPBL may be more effective than traditional learning or traditional PBL. There is limited evidence for other outcomes such as satisfaction, attitudes, cost effectiveness, and adverse effects. Most studies evaluated the use of digital technology for the presentation of problems as part of face-to-face DPBL and had unclear risk of bias. There is scope for the evaluation of digital technology in the delivery of other PBL components as well as the effectiveness of distance-based PBL.
Description of the digital education modalities.
MEDLINE (Ovid) search strategy.
Digital technologies used in the studies and associated learning types.
Characteristics and effects of the included studies.
Arena Blended Connected
digital problem-based learning
standardized mean difference
problem-based learning
randomized controlled trial
virtual patient
virtual reality
This review was conducted in collaboration with the Health Workforce Department at the World Health Organization. We thank Ms Ushashree Divakar and Ms Nuraini Nazeha for their help during the initial phase of the review and Dr Ram Bajpai for his statistical advice.
We would like to acknowledge funding support from the Lee Kong Chian School of Medicine, Nanyang Technological University Singapore (eLearning for health professional education grant).
LC and BK conceived the idea. BK and GD screened the articles and extracted the data. BK and LC wrote the manuscript. LC, GD, NS, MS, JR, NB, and JC provided insightful comments on the manuscript.
None declared.