This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
There is a pressing need to implement efficient and cost-effective training to address the worldwide shortage of health professionals. Mobile digital education (mLearning) has been mooted as a potential solution to increase the delivery of health professions education as it offers the opportunity for wide access at low cost and flexibility with the portability of mobile devices. To better inform policy making, we need to determine the effectiveness of mLearning.
The primary objective of this review was to evaluate the effectiveness of mLearning interventions for delivering health professions education in terms of learners’ knowledge, skills, attitudes, and satisfaction.
We performed a systematic review of the effectiveness of mLearning in health professions education using standard Cochrane methodology. We searched 7 major bibliographic databases from January 1990 to August 2017 and included randomized controlled trials (RCTs) or cluster RCTs.
A total of 29 studies, including 3175 learners, met the inclusion criteria. A total of 25 studies were RCTs and 4 were cluster RCTs. Interventions comprised tablet or smartphone apps, personal digital assistants, basic mobile phones, iPods, and Moving Picture Experts Group-1 audio layer 3 player devices to deliver learning content. A total of 20 studies assessed knowledge (n=2469) and compared mLearning or blended learning to traditional learning or another form of digital education. The pooled estimate of studies favored mLearning over traditional learning for knowledge (standardized mean difference [SMD]=0.43, 95% CI 0.05-0.80, N=11 studies, low-quality evidence). There was no difference between blended learning and traditional learning for knowledge (SMD=0.20, 95% CI –0.47 to 0.86, N=6 studies, low-quality evidence). A total of 14 studies assessed skills (n=1097) and compared mLearning or blended learning to traditional learning or another form of digital education. The pooled estimate of studies favored mLearning (SMD=1.12, 95% CI 0.56-1.69, N=5 studies, moderate quality evidence) and blended learning (SMD=1.06, 95% CI 0.09-2.03, N=7 studies, low-quality evidence) over traditional learning for skills. A total of 5 and 4 studies assessed attitudes (n=440) and satisfaction (n=327), respectively, with inconclusive findings reported for each outcome. The risk of bias was judged as high in 16 studies.
The evidence base suggests that mLearning is as effective as traditional learning or possibly more so. Although acknowledging the heterogeneity among the studies, this synthesis provides encouraging early evidence to strengthen efforts aimed at expanding health professions education using mobile devices in order to help tackle the global shortage of health professionals.
In 2013, the World Health Organization estimated that there was a shortage of 17.4 million health care workers worldwide: around 2.6 million doctors, and approximately 9 million nurses and midwives [
Digital education is a broad construct describing a wide range of teaching and learning strategies that are exclusively based on the use of electronic media and devices as training, communication, and interaction tools [
Digital education can entail various types of interventions that can be characterized in different ways: according to delivery tools, content, learning objectives, pedagogical approaches or settings of delivery. We categorized digital education according to the mode of delivery of digital education intervention and the pedagogical methods. Digital education includes, but is not limited to, offline and online computer-based education, serious gaming and gamification, massive open online courses, virtual reality environments, augmented reality, virtual patient simulations, psychomotor skills trainers and mobile digital education (mLearning) among others [
There is no uniformly accepted definition of mLearning. This lack of consensus not only arises from the rapid evolution of the field but also from ambiguity of the term “mobile.” Earlier definitions of mLearning were technocentric and only focused on the types of devices used, (eg, through a smartphone or tablet), or the situational context in which learning takes place (eg, on the way back home) [
mLearning is increasingly used in health professions education before (preregistration) and after qualification (postregistration), for example, as part of specialty training, continuous medical education or continuous personal development. In this review, we present the evidence collated on the use of mLearning in pre and postregistration health professions education. We considered eligible studies on candidates for, and holders of, the qualifications listed in the Health Field of Education and Training of the International Standard Classification of Education. We combine both the technocentric and the learner-centered approaches by defining handheld, mobile devices as being “small, autonomous, and unobtrusive enough to accompany us in every moment of our every-day life” [
Past reviews have underlined the potential of mLearning interventions but also stressed upon the need for further research and reviews on the topic [
The primary objective of this review is to evaluate the effectiveness of mLearning educational interventions for delivering preregistration and postregistration health professions education.
We adhered to the published protocol [
We developed a comprehensive search strategy for Medical Literature Analysis and Retrieval System Online (Ovid), EMBASE (Elsevier), Cochrane Central Register of Controlled Trials (Wiley), PsycINFO (Ovid), Educational Research Information Centre (Ovid), Cumulative Index to Nursing and Allied Health Literature (Ebsco) and Web of Science Core Collection (Thomson Reuters).
We searched reference lists of all the studies that we deemed eligible for inclusion in our review and relevant systematic reviews. We also searched the International Clinical Trials Registry Platform Search Portal and metaRegister of Controlled Trials to identify unpublished trials from and including 1990 to August 16, 2017.
We included RCTs and cluster RCTs. We excluded crossover trials because of a high likelihood of a carry-over effect. We included studies with students enrolled in either preregistration or postregistration health professions educational programs. We defined preregistration, undergraduate education or basic vocational training as any type of study leading to a qualification that (1) is recognized by the relevant governmental or professional bodies of the country where the studies were conducted and (2) entitles the qualification-holder to apply for entry-level positions in the health care workforce. Postregistration health profession educational programs were defined as any type of study after a qualification, which is recognized by the relevant governmental or professional bodies that enable the qualification holder entry into or continuation of work in the health care workforce in a more independent or senior role. Participants were not excluded based on age, gender, or any other sociodemographic characteristic.
We included studies in which mLearning interventions were used to deliver the learning content of the course. This includes studies where mLearning methods were the sole means by which the intervention was delivered or where mLearning methods were used in combination with traditional learning (ie, blended learning), as long as the contribution of the mLearning component to overall learning has been assessed.
mLearning interventions were defined as any teaching, learning and/or training intervention that was delivered through handheld mobile devices using wireless transmissions: third generation of mobile telecommunications technology, fourth generation of mobile telecommunications technology, global system for mobile communications, originally groupe spécial mobile (GSM), general packet radio services (GPRS), enhanced data rates for GSM evolution (EDGE or EGPRS), multimedia messaging service, short message service, universal mobile telecommunications system, wireless networking (Wi-Fi or any other wireless local area network) or long term evolution standard. Handheld mobile devices include but are not limited to mobile phones, smartphones, personal digital assistants (PDAs), tablets and Moving Picture Experts Group-1 audio layer 3 (MP3) players.
To be eligible for inclusion, studies have to report at least one of the following primary or secondary outcomes. The primary outcomes (measured using any validated or nonvalidated instruments) were the following: (1) students’ knowledge postintervention scores, (2) students’ skills postintervention, (3) students’ attitudes postintervention toward the mLearning intervention, education, or new clinical knowledge, and (4) learners’ satisfaction postintervention with the mLearning intervention.
Secondary outcomes were patient-related outcomes, change in clinical practices, economic aspects of the mLearning interventions (eg, cost-effectiveness, implementation cost, return on investment), changes in accessibility and/or availability of education and any adverse and/or unintended effects of mLearning interventions.
The search results from different electronic databases were combined in a single library and duplicates were removed. A total of 2 authors (GD and CKN) independently screened titles and abstracts of all records to identify potentially relevant studies. We retrieved full-text copies of those articles deemed potentially relevant. Finally, the same 2 reviewers independently assessed the full-text versions of the retrieved articles against the eligibility criteria. Any disagreements were resolved through discussion between the 2 screeners with a third review author (LTC) acting as an arbiter.
A total of 2 reviewers out of 4 (GD, CKN, LTC, and SN) independently extracted relevant characteristics related to participants, intervention, comparators, outcome measures, and results from all the included studies using a standard data collection form (see
A total of 2 authors out of 3 (GD, CKN, and LTC) independently assessed the risk of bias of RCTs and cluster-RCTs using the Cochrane risk of bias assessment tool [
For cluster RCTs, we also assessed the risk of these additional biases: recruitment bias, baseline imbalance, loss of clusters, incorrect analysis, and comparability with individually randomized trials. Judgments concerning the risk of bias for each study will be classified using “yes,” “no,” or “unclear,” indicating high, low, or unclear risk of bias, respectively. We incorporated the results of the risk of bias assessment into the review using a graph and a narrative summary.
We were unable to identify a clinically meaningful interpretation of effect size in the literature for digital education interventions. Therefore, in line with other studies in the field, we present outcomes using postintervention standardized mean difference (SMD) and interpret the effect size using Cohen’s ‘rule of thumb’ (ie, with 0.2 representing a small effect, 0.5 a moderate effect, and 0.8 a large effect) [
We aimed to present uniform postintervention data (ie, SMDs for continuous outcomes with their respective confidence intervals) to ensure consistency and comparability of data. For the meta-analysis, we used a random-effects model as different scales were used in different studies. We used the generic inverse variance method to combine cluster and noncluster RCTs of continuous outcomes. The effect estimates with corresponding 95% CIs for each study as well as a pooled effect size with 95% CI were displayed in the forest plots. We performed meta-analysis using Review Manager 5.3 (Cochrane Library Software, Oxford, UK) [
We developed a preliminary synthesis by grouping the included studies by the type of interventions and comparators used:
mLearning versus traditional learning
Blended learning versus traditional learning
mLearning versus other forms of digital education
We prepared a
Our search strategy retrieved 30,532 unique references, of these, 29 studies fulfilled inclusion criteria [
We included 29 studies involving 3175 participants [
A total of 25 out of 29 studies were RCTs, and the remaining 4 studies were cluster RCTs [
Study flow diagram. RCT: randomized controlled trial.
Number of studies by year of publication and mLearning device. PDA: personal digital assistant.
Country of origin of included studies for low- and middle-income and high-income countries separately. WHO: World Health Organization.
Forest plot for the knowledge outcome (postintervention).
A total of 20 studies (n=2469) assessed knowledge posttest scores as a primary outcome [
A total of 11 studies compared mLearning methods versus traditional learning, assessing knowledge gain postintervention [
The pooled estimate of the studies favored mLearning over traditional learning in terms of postintervention knowledge scores (SMD=0.43, 95% CI 0.05-0.80, N=11 studies, low-quality evidence; see
A total of 6 studies compared blended learning methods (mLearning plus traditional learning) with traditional learning to assess knowledge gain postintervention [
There was no difference between blended learning and traditional learning groups in terms of postintervention knowledge scores (SMD=0.20, 95% CI –0.47, 0.86, N=6 studies, low-quality evidence; see
A total of 2 studies compared one form of mLearning with another form of mLearning to assess knowledge gain postintervention [
One study (100 participants) compared mLearning with another form of digital education to assess knowledge gain postintervention [
A total of 14 studies assessed skill acquisition of mLearning interventions compared with various controls and included a total of 1097 participants [
A total of 5 studies compared mLearning methods with traditional learning, assessing skill acquisition postintervention [
The pooled estimate of the studies favored mLearning over traditional learning in terms of postintervention skill acquisition (SMD=1.12, 95% CI 0.56 to 1.69, N=5 studies, moderate quality evidence; see
A total of 8 studies compared blended learning methods versus traditional learning, assessing skill acquisition postintervention [
The pooled estimate of the studies favored blended learning over traditional learning in terms of postintervention skill acquisition scores (SMD=1.06, 95% CI 0.09-2.03, N=7 studies, low-quality evidence; see
We are uncertain about the effect of 1 study (183 participants) because of incomparable outcome data [
Forest plot for the skills outcome (postintervention).
One study (63 participants) included 4 groups receiving varying forms of an mLearning intervention, as was described earlier in the review [
A total of 5 studies assessed participants’ attitudes following the mLearning or blended learning intervention and included a total of 440 participants [
A total of 2 studies comparing mLearning methods versus traditional learning assessed participants’ attitude postintervention [
The pooled estimate of the studies favored mLearning over traditional learning in terms of postintervention attitudes (SMD=0.51, 95% CI 0.20-0.81, N=2 studies, low-quality evidence, I2=0%).
One study (72 participants) reported that participants in the blended learning group felt more confident in their ability to identify the anatomical structures postintervention compared with traditional learning [
A total of 4 studies [
A total of 2 studies assessed participants’ postintervention satisfaction scores in the mLearning interventions compared with traditional learning [
There was no difference between mLearning and traditional learning groups in terms of postintervention satisfaction (SMD=0.39, 95% CI –0.29, 1.06, N=2 studies, very low-quality evidence). There was a substantial amount of heterogeneity in the pooled analyses (I2=79%).
No study assessed participants’ postintervention satisfaction scores in both the blended learning intervention group and the traditional learning group. A total of 2 studies [
A total of 2 studies [
One study reported on patient-related outcomes [
A total of 2 studies reported on changes in clinical practices/behaviors [
None of the included studies reported any reported any adverse and/or unintended effects of the mLearning interventions.
We were unable to assess the changes in accessibility and/or availability of education because of limited information in the included studies.
There was not sufficient data to allow sensitivity analyses to be conducted.
There were not enough comparisons to carry out a formal assessment of publication bias.
As presented in the risk of bias summary (
More than four-fifth of studies (86%) did not provide information on the method of randomization and sequence allocation. The majority of studies (72%) reported the use of outcome measures to blind assessors or used self-report questionnaires or MCQs in the outcome assessment, which we believed did not require blinding, and thus these studies were judged to be of a low risk of bias. The remaining studies (28%) were judged to be of an unclear risk of bias because of a lack of information. A total of 3 studies (10%) were judged to be of a high risk of attrition bias as these studies had a high dropout rate (35%-73%) and/or no reasons for missing data were reported and/or lacked intention to treat analysis for the missing data, a further 6 studies (20%) were judged as unclear because of a lack of information. One study (3%) was judged to be of a high risk of bias for selective reporting, as an outcome stated in the methods section was not reported in the results section, the rest of the studies were judged to be of a low risk of bias for selective reporting. A total of 11 studies (38%) did not provide any information on a baseline assessment and were judged to be of an unclear risk of other bias.
In the clustered RCTs, only 1 study accounted for clustering reporting both individual level and cluster levels results [
Risk of bias summary: review authors’ judgments about each risk of bias item across all included studies.
Learning is changing and is increasingly becoming mobile. Our findings suggest that mLearning interventions are equivalent or possibly superior to traditional learning methods for improving knowledge and skill in pre and postregistration health professions education (see the summary of findings
With 21 of the 29 included studies (72%) published between 2014 and 2017, it’s clear that mLearning is an emerging educational strategy. The remaining 8 studies were published between 2006 and 2013, with no studies published before 2006, further highlighting the modern nature of this approach to health professions education and its relevance. The novel nature of mLearning, coupled with the continuing rapid advancements in mobile technology will likely see mLearning continue to evolve, as it has with the studies included in this review. This evolution is graphically illustrated in
The majority of studies focused on preregistration medical and nursing students and residents undertaking specialty training. A smaller number of studies focused on practicing physicians [
As mLearning may involve the use of new technology, which can entail significant financial investment, the cost of such an introduction, especially in low-income settings, is a key factor when deciding to adopt new mediums for education. Unfortunately, only 2 studies, with conflicting results, performed an economic analysis comparing mLearning with traditional learning methods. As a result, there is limited evidence to draw any definitive conclusions on the costs and cost-effectiveness of mLearning. Similarly, there was little information on patient-related outcomes or changes in clinical behaviors/practices. A further aspect of mLearning that needs addressing is the impact of mLearning on the adverse/unintended effects of mLearning on patients as well as the learner.
For the majority of the studies, the intervention was introduced as supplementary learning to their standard education, and therefore questions remain as to how effective mLearning would be when introduced to a formal curriculum. In addition, the interventions were often introduced on an optional basis, where the use and intensity of the intervention were dependent on the individual and exact exposure and effect of the intervention are unknown.
Only 6 studies mentioned the use of learning theories to inform their instructional design. This indicates a major limitation of mLearning within health professions education. There is a lack of theoretical frameworks guiding effective instructional design so that learning pathways using adopted technologies could be delineated clearly [
This review provides the most up-to-date evidence on the effectiveness of mLearning in health professions education and is supported by a comprehensive search strategy and the robust methodology that was applied at each stage of the screening, data extraction, and assessment of the evidence. This is illustrated by the larger body of evidence gathered in this review compared with previous reviews [
Common biases include study eligibility criteria, identification and selection of studies, data extraction, and study appraisal. We tried to minimize or eliminate these biases in this review by adopting a variety of quality checks. We prespecified the eligibility criteria of the studies to be included in the review, and those were clearly defined in the protocol published before carrying out the review. This measure ensured that decisions on which studies were to be included were consistent and not based on characteristics of potentially eligible studies. The search strategy was devised and conducted by experienced librarians including all appropriate databases. As the search strategy was devised for a general project on digital education, it included Medical Subject Headings terms for several different digital education modalities. The search strategy retrieved a very large number of references. Titles and abstracts were screened independently by a team of reviewers and full-text inclusion assessment involved at least 2 reviewers. Furthermore, lead authors of the reviews of the other digital education modalities transferred studies from their reviews to other more appropriate reviews if necessary. Data availability bias may occur if some data are unavailable in the included studies and their unavailability is related to the study results. As with publication bias, this situation may lead to unrepresentative data and toward a false favorable effect. We contacted authors of studies with missing data or no data on specific outcomes to ask for those or to ask clarifications. Overall the risk of bias for most studies was judged to be high (because of a lack of information), with some instances of a high risk of bias for sequence generation, attrition, and reporting bias identified.
Reasons for downgrading the evidence included inconsistency, that is, high heterogeneity/differences in the direction of effect, with high the I2 values reported for each of the knowledge and skill comparisons. We did not identify a sufficient number of studies within the review comparisons to allow for the performance of subgroup analyses, which were prespecified in the protocol. With the number of studies of mLearning in health professions education continuing to rise over time, future reviews will be able to perform more focused subgroup analyses.
The review identified gaps in evidence, which if addressed, would provide more conclusive evidence on the effectiveness and cost-effectiveness of mLearning. Further research should do the following: assume validated and standardized outcome measures, use adequately powered trials, ensure that participants are adequately trained and empowered to use the mLearning devices, be adequately reported to allow independent replications, shift toward competency-based assessments, include theoretical underpinning in instructional design, include participants from other health professions such as pharmacy and occupational therapy; be conducted in more LMICs, incorporate a more in-depth study of the various aspects of mLearning (eg, interactivity, feedback) and how each specific component affects study outcomes, provide information about the effects of mLearning on patient outcomes, provide information on cost and cost-effectiveness of mLearning, provide information on potential unintended effects of mLearning, and include both short-term and retention (follow up) outcome data.
mLearning is a novel educational strategy that is rapidly developing in the field of health professions education. The synthesis of data in this review shows that mobile learning is at least as or potentially more effective than traditional learning. However, the effectiveness of mLearning in health professions education is not certain because of the lack of validated and standardized outcome measures, and heterogeneity between both interventions and outcome assessments. Furthermore, there is a need for research to expand to the realm of cost-effectiveness, to fully understand the value of mLearning in health professions education. Further research is necessary to conclusively evaluate the effectiveness and cost-effectiveness of mLearning.
MEDLINE (Ovid) search strategy.
Data extraction form.
Characteristics of included studies.
Summary of findings table for mLearning versus traditional learning.
Summary of findings table for blended learning vs traditional learning.
Risk of bias graph.
Risk of bias for cluster randomized controlled trials.
Geriatric Assessment Tool
general packet radio services
global rating scale
global system for mobile communications
low- and middle-income countries
Moving Picture Experts Group-1 audio layer 3
multiple choice questionnaire
personal digital assistant
randomized controlled trial
standardized mean difference
short message service
This review was conducted in collaboration with the Health Workforce Department at the World Health Organization. We would like to thank Mr Carl Gornitzki, Ms GunBrit Knutssön and Mr Klas Moberg from the University Library, Karolinska Institutet, Sweden, for developing the search strategy, and the peer reviewers for their comments. We also would like to thank Dr Ram Chandra Bajpai for his suggestions in data analysis. The authors gratefully acknowledge funding from the Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, eLearning for health professionals education grant, and from National Healthcare Group, Singapore.
LTC conceived the idea for the review. GD and CN wrote the review. LTC provided methodological guidance, drafted some of the methodology-related sections and critically revised the review. SN, RA, GCYL provided comments on and edited the review.
None declared.