Background: There is a lack of evidence in the literature regarding the learning outcomes of immersive technologies as educational tools for teaching university-level health care students.
Objective: The aim of this review is to assess the learning outcomes of immersive technologies compared with traditional learning modalities with regard to knowledge and the participants’ learning experience in medical, midwifery, and nursing preclinical university education.
Methods: A systematic review was conducted according to the Cochrane Collaboration guidelines. Randomized controlled trials comparing traditional learning methods with virtual, augmented, or mixed reality for the education of medicine, nursing, or midwifery students were evaluated. The identified studies were screened by 2 authors independently. Disagreements were discussed with a third reviewer. The quality of evidence was assessed using the Medical Education Research Study Quality Instrument (MERSQI). The review protocol was registered with PROSPERO (International Prospective Register of Systematic Reviews) in April 2020.
Results: Of 15,627 studies, 29 (0.19%) randomized controlled trials (N=2722 students) were included and evaluated using the MERSQI tool. Knowledge gain was found to be equal when immersive technologies were compared with traditional learning modalities; however, the learning experience increased with immersive technologies. The mean MERSQI score was 12.64 (SD 1.6), the median was 12.50, and the mode was 13.50. Immersive technology was predominantly used to teach clinical skills (15/29, 52%), and virtual reality (22/29, 76%) was the most commonly used form of immersive technology. Knowledge was the primary outcome in 97% (28/29) of studies. Approximately 66% (19/29) of studies used validated instruments and scales to assess secondary learning outcomes, including satisfaction, self-efficacy, engagement, and perceptions of the learning experience. Of the 29 studies, 19 (66%) included medical students (1706/2722, 62.67%), 8 (28%) included nursing students (727/2722, 26.71%), and 2 (7%) included both medical and nursing students (289/2722, 10.62%). There were no studies involving midwifery students. The studies were based on the following disciplines: anatomy, basic clinical skills and history-taking skills, neurology, respiratory medicine, acute medicine, dermatology, communication skills, internal medicine, and emergency medicine.
Conclusions: Virtual, augmented, and mixed reality play an important role in the education of preclinical medical and nursing university students. When compared with traditional educational modalities, the learning gain is equal with immersive technologies. Learning outcomes such as student satisfaction, self-efficacy, and engagement all increase with the use of immersive technology, suggesting that it is an optimal tool for education.
Educational technology is changing the way in which we learn today, and its purpose is to ultimately improve education [, ]. The addition of educational technology to a curriculum needs to be developed and guided by informed, evidence-based research. Educational technology includes instructional software such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), known collectively as immersive technology [ ]. Immersive technologies should be built around effective teaching methods that provide an appropriate learning method and learning outcome [ ]. Immersive technologies are thought to provide pedagogy based on the constructivist theory and experiential learning, creating an environment that aids visual learners and enables students to learn by doing, develop creativity, and increase understanding of invisible concepts [ ]. The Association for Medical Education in Europe has previously published guidance on e-learning in medical education: “Designs for effective medical e-learning, therefore, need to mirror the dynamics and details of real-world practice as well as affording effective learning opportunities” [ ].
Immersive technologies are defined as devices that provide sensory stimuli to provide a sense of realism and immersion in the interactions with the computer-generated world . VR is a technology that allows the user to explore and manipulate computer-generated real or artificial 3D multimedia sensory environments in real time. It dates back to 1956, when Morton Heilig, a cinematographer, developed the Sensorama, a display box first used for background scenes in the Hollywood motion picture industry. This was the first head-mounted display to be developed. In the mid-1960s, Ivan Sutherland, an American Computer Scientist, went on to develop the concepts of VR further. He described The Ultimate Display, a VR system that could simulate reality [ ], and his paper described core concepts that are the foundation of VR use today. Owing to the heterogeneity of the terminology used to describe VR, we can characterize VR as a “collection of hardware such as Personal Computer (PC), HMDs and tracking sensors, as well as software to deliver an immersive experience” [ ]. In comparison, AR is an interactive experience of a real-world environment where the objects that reside in the real world are augmented by computer-generated perceptual information. Historically, the development of AR started in the 1960s; however, the term AR was not established until 1990. Although VR and AR share many technical aspects, the main difference is that AR does not construct a fully artificial environment and simply overlays computer-generated images onto images of the real world. Therefore, it uses machines that allow a physical view of the surrounding environment to be visible but enhanced with virtual images [ ]. Finally, MR is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects coexist and interact in real time [ ].
To date, there has been a multitude of publications detailing the development and implementation of immersive tools, in addition to demonstrating the benefits of VR, AR and MR technology in medical, nursing and midwifery education [- ]. This technology is thought to provide increased engagement and understanding during learning coupled with feedback mechanisms and design capabilities of varying difficulty levels [ ]. In addition, it facilitates practice without the risk of human harm and also helps build professional skills and teamwork [ , ]. However, there is a paucity of evidence on the learning outcomes of these innovative educational tools.
As outlined by the Digital Health Education Collaboration, there is a need for a strong evidence base to guide the development of immersive educational tools so that the learning goals and outcomes are in line with national and international standards . There have been an increasing number of systematic reviews documenting the use, application, and effectiveness of VR, AR, and MR in an effort to establish an evidence-based network of research for use in medical education. However, the results have been mixed; including a systematic review that looked at the effectiveness of AR in medical education which found that there was insufficient evidence to recommend its implementation into the curriculum. Similarly, another review looked at serious games used in medical education and found that the evidence was moderate to support the use of immersive technology, stating that it should not replace traditional learning tools [ - ]. Immersive technologies are used mainly as educational tools for complex topics such as anatomy and embryology and are thought to enhance the learning experience [ , ]. Are VR, AR, and MR as effective in delivering knowledge as well as an enhanced learning experience in comparison with traditional teaching tools such as 2D didactic presentations?
Therefore, the aim of this systematic review is to assess the learning outcomes of VR, AR, and MR across 3 health care student disciplines—medicine, nursing, and midwifery education—compared with traditional learning modalities. The learning outcomes include knowledge, skill development, and the learning perceptions of students, including satisfaction and self-confidence in learning along with engagement and motivational factors.
Purpose and Protocol
A systematic review of the available scientific literature was conducted to assess the learning outcomes associated with the application of VR, AR, and MR as educational tools compared with traditional learning modalities for medical, nursing, and midwifery students in preclinical university education. The review protocol was registered with PROSPERO (International Prospective Register of Systematic Reviews) in April 2020 (CRD42020154598). The search results were reported according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines  and the Cochrane Collaboration guidelines [ ].
The eligibility criteria were based on the Population, Intervention, Comparison, and Outcomes criteria. The population selected for this review included preclinical university students enrolled in three educational disciplines: medicine, nursing, or midwifery courses only. Randomized controlled trials (RCTs) that implemented VR, AR, or MR technology in comparison with a control method were included. Owing to the heterogeneity of the definitions surrounding VR applications, we restricted inclusion to interactive 3D models requiring a headset, virtual patients (VPs), or VR learning environments. The primary outcomes included knowledge and reference to the learning experience, which involved engagement, satisfaction, and perceived learning experience. Only studies published in English were included.
A large-scale search was undertaken because of the wide use of the various terminology to describe VR, AR, and MR and the technology surrounding their use in health care student education. The following method was used to identify empirical studies for inclusion in the systematic review. We conducted a comprehensive computerized database search of full-text articles published in English. Only RCTs assessing learning outcomes using VR, AR, or MR technologies in comparison with traditional learning models were included. The reason for this was that we wanted to review the learning outcomes of immersive technologies compared with traditional learning outcomes, including knowledge and learner experience. The fundamental study design of an RCT requires a control and an intervention group; therefore, we selected these types of studies for this review. Searches were conducted with predefined search terms () using the following electronic databases: PubMed, Embase, Web of Science, CINAHL, and ERIC. Medical Subject Headings terms included virtual reality, augmented reality, educational technology, imaging, three dimensional, education, and teaching materials. Search terms were connected using Boolean operators AND and OR to capture all relevant article suggestions. The latest search was conducted on March 8, 2021.
Databases were downloaded to EndNote reference manager software (Clarivate Analytics), which recorded citations and identified duplicates. A spreadsheet was created to record decisions and comments. Screening of articles was conducted by 2 researchers (GR and SC) in an unblinded, standardized approach (independently and in parallel). Titles and abstracts of studies sourced from electronic databases were reviewed according to the inclusion and exclusion criteria previously described. Subsequently, full texts of the included articles from the initial screening process were reviewed for eligibility according to the described inclusion and exclusion criteria. Differences of opinion were resolved through conversations between the reviewers. A schematic stepwise algorithm for the search strategy is shown in.
Microsoft Excel was used to build a data extraction form, which was divided into three categories: (1) study identification, (2) analysis of learning outcomes, and (3) study design. The first section included bibliographic information, the country of origin of the study, and a demographic description of the participants. The second section examined learning outcomes related to teaching strategies, relationships between technologies, and learning objectives. The third section evaluated the methodological quality of the study design.
Study Quality Assessment
We used the Medical Education Research Study Quality Instrument (MERSQI) to evaluate the study design of the RCTs . The MERSQI is divided into several domains, including evaluation of study design, sampling, data type, validity, data analysis, and outcomes. The learning outcomes are based on the hierarchy of educational outcomes by Kirkpatrick and Kirkpatrick [ ], which adopts a constructional framework using a 4-level model for evaluating educational effectiveness. The first level describes the participants’ perception of the learning experience; knowledge, skills, and attitudinal change are assessed in the second level; changes in behavior are evaluated in the third level; and changes in health care or patient outcomes are evaluated in the fourth level.
A narrative review of the results reported in the included studies on learning outcomes was conducted. The data in the final included studies did not allow for a formal meta-analysis as the studies were not sufficiently homogenous, given the stated research question and the use of different technologies and educational topics.
We identified 15,627 articles from the primary database search. After duplicates were removed, there were 86.63% (13,538/15,627) of articles left for abstract review. Abstracts were screened and, of those 13,538 articles, 179 (1.32%) remained for a full paper review. Of those 179 articles, 150 (83.8%) were excluded, leaving 29 (16.2%) full papers for study inclusion. Details of the study selection process are displayed in. In total, 29 RCT studies (N=2722) were included in this review. All studies were conducted in the past 10 years, with most studies (18/29, 62%) published within the past 3 years.
In total, 2722 students participated in the 29 RCTs. Of the 29 articles, 19 (66%) included medical students (1706/2722, 62.67%), 8 (28%) included nursing students (727/2722, 26.71%), and none of the studies involved midwifery students. Approximately 7% (2/29) of studies included both medical and nursing students (289/2722, 10.62%). The following disciplines were used to test the immersive technologies: anatomy, basic clinical and history-taking skills, neurology, respiratory medicine, acute medicine, dermatology, communication skills, internal medicine, and emergency medicine. A full list of the RCTs, basic demographic details, and immersive technology applications included in this review is outlined in.
|Author||Setting||Application detail||Sample size, N||Purpose||Outcome|
|Seifert et al ||Germany||VPa cases (Moodle learning management system)||40||VP—basic clinical skills||Similar levels of long-term knowledge gained; participants assessed the learning experience and the comprehensibility of the seminars as either very good or good|
|Wang et al ||New Zealand||3D visualizer software (preloaded 3D hologram) on Microsoft HoloLens device||52||Anatomy teaching||There was no difference in knowledge acquisition between groups; only MRb group demonstrated higher retention in nominal and spatial types of information; increased engagement in 3DMc and MR group|
|Rossler et al ||United States||Virtual Electrosurgery Skill Trainer developed by the National Institutes of Health||20||Fire safety knowledge||No differences in knowledge; intervention group participants were noted to meet performance criteria for their assigned role in their perioperative team|
|Lombardi et al ||United States||Virtual heart activities using physiology software programs (Practice Anatomy Lab, Pearson Education, and Interactive Physiology)||29||Anatomy teaching||Plastic model group achieved significantly higher overall scores on initial and follow-up exams; attitude surveys demonstrated a higher preference for organ dissection|
|Padilha et al ||Portugal||Body Interact (simulation with VPs)||42||Respiratory medicine||Improved knowledge and higher levels of learning satisfaction in the intervention group; no statistically significant differences in self-efficacy perceptions|
|Blanie et al ||France||VP cases—LabForSIMS (simulation center) and a software designer (Interaction Healthcare)||146||Basic clinical skills||No significant educational difference was found; satisfaction and motivation were found to be greater with the use of SGd|
|Liaw et al ||Singapore||VP simulation—eRAPIDS, developed at the National University of Singapore||57||Clinical deterioration||No difference in knowledge acquisition; VP was rated positively|
|Menzel et al ||United States||Second Life (Linden Lab) virtual simulation environment (WALDe Island)||51||Cultural attitudes||No statistically significant differences between the learning formats|
|Gananasegaram et al ||Canada||Campbell’s 3DM of the inner ear—publicly available data sets displayed on Microsoft HoloLens||29||Anatomy teaching||No difference in knowledge acquisition; HGf group rated higher for overall effectiveness, ability to convey spatial relationships, and learner engagement and motivation|
|Liaw et al ||Singapore||VRg (no details)||198||VP to teach MDTh rounds||Increased levels of self-efficacy and attitudes toward interprofessional team care|
|Moro et al ||Australia||Microsoft HoloLens, 3D Studio Max (Autodesk Inc), Unity 3D (Unity Technologies), Vuforia v5 plug-in for Unity (PTC Inc), Samsung Galaxy Tab 3 (Samsung Electronics), Visual Studio v2019||40||Physiology and brain anatomy||No difference in knowledge test scores; significant increase in dizziness using the HoloLens|
|Stepan et al ||United States||VR model of brain anatomy—brain CTi scans and MRIsj, Surgical Theater, Oculus Rift VR system (Oculus VR)||66||Cerebral anatomy||No difference in anatomy knowledge; VR group found learning experience to be significantly more engaging, enjoyable, useful, and motivating|
|Hu et al ||Taiwan||Anatomy Master module of Medical Holodeck||101||Anatomy teaching||Significant improvement in ultrasound task performance and ultrasonographic image identification MCQk tests in the VR group|
|Engum et al ||United States||CathSim Intravenous Training System (CathSim) developed by HT Medical (Immersion)||93||Intravenous catheter training||Significant improvement in cognitive gains, student satisfaction, and documentation of the procedure with the traditional laboratory group compared with the computer catheter simulator group|
|Berg et al ||Norway||VR application developed by the authors with hired help for programming (Unity 2018.30f2) and video of the VR features||289||ABCDE basic resuscitation skills||Noninferiority of learning modality; more students in VR group reported liking the way they practiced and that it was a good way to learn; VR group scored high on the System Usability Scale|
|Kiesewetter et al ||Germany||VR learning environment CASUS||142||VP to teach clinical skills||Case formats with a VP did not affect knowledge gain or diagnostic accuracy |
|Schoeb et al ||Germany||Instructions for catheterization displayed on Microsoft HoloLens||164||Catheter training||MR group had significantly better learning outcomes |
|Noll et al ||Germany||ARl mobile app, iPhone operating system (iOS, Apple Inc)–based app mArble Derma (m-ARBLE-dermatology)||44||Dermatological teaching||No difference in outcomes between groups |
|Liaw et al ||Singapore||3D virtual hospital developed—CREATIVE||120||Interprofessional skill training||No difference between groups in communication performance scores |
|Ienghong et al ||Thailand||3D USSm images played on the downloaded phone app and AR||46||Emergency ultrasound skills||Better performance scores in VR flash card group |
|Sobocan et al ||Slovenia||VP—no detail||34||Internal medicine skills||No difference in exam performance between groups|
|Kockro et al ||Switzerland||Virtual 3DM developed from MRI and CT scans and DextroBeam system (Bracco Advanced Medical Technologies)||169||Neuroanatomy||There were no significant differences in knowledge scores; participants rated the 3D method as superior to 2D teaching methods in four domains: spatial understanding, application in future anatomy classes, effectiveness, and enjoyableness |
|Nickel et al ||Germany||Computer-based TMn developed using the open-source Medical Imaging Interaction Toolkit software developed by the German Cancer Research Center||410||Liver anatomy||3D group had higher scores, and participants had a preference for 3D training |
|Berg et al ||Norway||ABCDE resuscitation application developed with help from hired programmers for Unity using Oculus Quest software (Oculus)||289||VP to teach clinical skills||Group self-practice of the ABCDE approach in multiplayer, immersive, interactive VR application was noninferior to practice with physical equipment |
|Bogomolva et al ||The Netherlands||DynamicAnatomy AR application developed at the Department of Anatomy and Embryology at the Leiden University Medical Centre for Innovation||60||Anatomy teaching||No significant differences in knowledge scores |
|O’Rourke et al ||United States||VP model simulation developed with a real patient in real time||60||Clinical skill—breaking bad news||No significant between-group differences on the POMSo 2 or salivary cortisol concentration following the SPp interaction; students had similar emotional and behavioral responses when delivering bad news to SP or vSPq |
|Du et al ||Taiwan||3DMs—Autodesk 3DS Max software (Autodesk Media and Entertainment) and Unity 3D developed into a VR gaming system for HTC Vive||18||Anatomy teaching||No significant differences in MCQ scores between groups; VR groups scored highly on the interest, competence, and importance subscales of the IMIr; both VR groups considered the system to be fun and beneficial to their learning; MPs group reported higher stress levels|
|Maresky et al ||Canada||Using TeraRecon (TeraRecon, Inc), Slicer, and The Body VR: Anatomy Viewer private beta version (The Body VR LLC) together with software provided by Sharecare VR (Sharecare Reality Lab)||42||Cardiac anatomy||VR group scored higher on knowledge|
|Issleib et al ||Germany||VR software developed in cooperation between the University of Hamburg and VIREEDt—VR-BLSu course (using the Laerdal (mannequin)||160||Resuscitation training||Classic BLS training with a seminar and training sessions seemed superior to VR in teaching technical skills |
aVP: virtual patient.
bMR: mixed reality.
c3DM: 3D model.
dSG: simulation by gaming.
eWALD Island named for Lillian Wald, a public health nursing pioneer.
gVR: virtual reality.
hMDT: multidisciplinary team.
iCT: computed tomography.
jMRI: magnetic resonance imaging.
kMCQ: multiple-choice question.
lAR: augmented reality.
mUSS: ultrasound scan.
nTM: teaching module.
oPOMS: Profile of Mood States.
pSP: simulated patient.
qvSP: virtual simulated patient.
rIMI: Intrinsic Motivation Inventory.
tVIREED: Virtual Reality Education (medical virtual reality education platform).
uBLS: basic life support.
Approximately 76% (22/29) of authors used VR applications, which included virtual simulation scenarios and VPs. Approximately 17% (5/29) of articles used AR applications, which involved using the Microsoft HoloLens headset and mobile phone apps. There were 7% (2/29) of studies that used MR.
Of the 29 articles retrieved, 28 (97%) were from high-income countries and 1 (3%) was from an upper–middle-income country. Most studies originated in the United States (6/29, 21%) and Germany (6/29, 21%), followed by Singapore (3/29, 10%), Canada (2/29, 7%), Norway (2/29, 7%), Taiwan (2/29, 7%), Australia (1/29, 3%), France (1/29, 3%), the Netherlands (1/29, 3%), New Zealand (1/29, 3%), Portugal (1/29, 3%), Slovenia (1/29, 3%), Switzerland (1/29, 3%), Thailand (1/29, 3%), and Turkey (1/29, 3%).
Primary and Secondary Learning Outcomes
Learning outcomes were reported in all studies, including outcomes for both knowledge and the participants’ learning experience. Knowledge was assessed via multiple-choice question tests, single best answer tests, general clinical knowledge tests, open-ended style tests, or objective structured clinical examinations. Of the 29 studies, 12 (41%) studies evaluated knowledge using multiple-choice question tests. Approximately 31% (9/29) of studies evaluated knowledge via general clinical knowledge–based tests that used a variety of validated questionnaires such as the Maastricht Assessment of Simulated Patients questionnaire and the Attitudes toward Poverty scale questionnaire [, ]. Approximately 10% (3/29) of studies assessed knowledge using open-ended style exam questions. Approximately 3% (1/29) of studies evaluated knowledge via an objective structured clinical examination–based exam. Of the 29 studies, 1 (3%) evaluated knowledge via a single best answer test, 1 (3%) did so via interprofessional skill assessment, and 1 (3%) did not specify the type of evaluation test. Approximately 90% (26/29) of studies reported on satisfaction, attitudes, perceptions, opinions, and general facts regarding the technology used. The learning experience was evaluated in various ways, including validated and nonvalidated instruments on satisfaction, engagement, and perceived learning outcomes. Approximately 3% (1/29) of studies used psychometric testing to evaluate the learning experience [ ]. Another study measured salivary cortisol levels before and after the intervention to evaluate whether delivering bad news via a real simulated patient or a virtual simulated patient evoked the same psychological stress [ ]. Approximately 66% (19/29) of studies used validated scales to assess the learning experience, namely, the Intrinsic Motivation Inventory [ ], General Self-Efficacy Scale, Learner Satisfaction with Simulation Scale [ ], and Diagnostic Thinking Inventory [ ]. A full list of the validated scales used in the RCTs in this review is provided in [ , , , - , , , - , , - ]. Approximately 7% (2/29) of studies reported on behaviors as an outcome, and none of the studies reported patient or health care outcomes [ , ]. Only 3% (1/29) of studies reported on adverse health outcomes as part of their secondary outcomes [ ].
Study Quality Assessment
The MERSQI scale scores ranged from 10 to 15, with a mean score of 12.64 (SD 1.6). The median was 12.5, and the mode was 13.5. The mean (SD) domain and item scores are illustrated in. The mean domain scores were highest for study design (mean 1, SD 0), data analysis (mean 0.7, SD 0.46), and outcomes (mean 0.6, SD 0.21). The lowest-scoring domains included type of data (mean 0.5, SD 0), sampling (mean 0.3, SD 0.15), and validity of the evaluation instrument (mean 0.3, SD 0.27). All articles used an RCT (29/29, 100%) study design, which meant that all studies obtained the highest score in this domain.
Of the 29 studies, 1 (3%) was a double-center RCT, and the remaining 28 (97%) were conducted at a single site. In relation to participant response rate, 93% (27/29) of the studies had a high response rate of >75%.
The authors used appropriate statistical analysis in all studies according to the MERSQI . In relation to the validity of the evaluation instrument, 66% (19/29) of studies used validated instruments and scales to assess learning outcomes, including satisfaction, self-efficacy, engagement, and perceptions of the learning experience.
Finally, according to classification using the criteria by Kirkpatrick  in the MERSQI scale, 97% (28/29) of studies assessed knowledge as the primary outcome, and 62% (18/29) of studies used a pre- and postlearning experience knowledge assessment.
The aim of this systematic review was to assess the current educational role of immersive technology in medical, midwifery, and nursing education at the university level compared with traditional learning modalities. Second, the review evaluated the associated learning outcomes of VR, AR, and MR and how they were evaluated. A total of 29 RCTs were selected for this review. The main findings of this study indicate that knowledge gain is equal when using VR, AR, or MR in health care student education compared with traditional learning modalities. In addition, VR, AR, and MR provide an enhanced learning experience based on the secondary outcomes of the studies included in this review [, , , , , , ]. This supports the current evidence that immersive educational technology is a useful and valuable learning tool in medical and nursing preclinical university education. The most common form of immersive technology used was VR. The favored use of VR may be due in part to the widespread availability of cost-effective 3D software and web-based material for the development of anatomy tools [ , ]. Comparators were present in every study and ranged from 2D didactic presentations to textbooks. All the studies retrieved involved medical and nursing students in preclinical university education. Interestingly, there were no studies involving midwifery students. Although there are studies involving midwifery students and immersive technologies in the literature, this review found no RCTs within this domain.
The MERSQI tool provided a standardized approach to evaluating the methodological quality of the studies included in this review , which resulted in a moderate level of evidence for these studies. The MERSQI tool also allowed us to classify the learning outcomes of the RCTs clearly, with knowledge identified as the most common primary outcome. Secondary outcomes were identified and included attitudes and opinions related to the learner experience (ie, satisfaction, self-efficacy, engagement, and perceptions of the learning experience). All studies were conducted in upper–middle-income or high-income countries, which may be because of the availability of more funding to support the purchase of such equipment. The articles identified for this review were published within the past 10 years, with most (19/29, 66%) being published in the past 3 years. This highlights the recent rise in interest in immersive technologies for use in preclinical university medical and nursing education. The International Data Corporation has forecasted the shipment of 36.7 million VR headset units by 2023 [ ].
Anatomy was the most common topic taught with immersive technologies, which may be because of the ability of immersive technologies to enhance the understanding of complex body processes that cannot be visualized . Subjects such as anatomy and embryology require students to translate 2D images into 3D concepts, which can be a cognitive challenge for those who have difficulty visualizing this transformation of images [ , ]. Students may also mentally rotate complex structures inaccurately, leaving the mind to fill the gaps in any missing structures [ ]. The introduction of immersive technology as an educational tool could automatically standardize this process for students and enhance the understanding of certain subjects such as anatomy and embryology. Similarly, midwifery education requires an understanding of complex concepts, such as fetal development in the womb. Therefore, this would be an important area for future research comparing traditional learning tools with immersive technologies.
VPs were developed in 41% (12/29) of the studies in this review. VPs play a key role in learning basic clinical skills for both nursing and medical education. In addition, they create an environment for repeated practice in a safe space without harm to patients [, ]. Traditionally, clinical education relies on practicing diagnostic, therapeutic, and procedural skills on live patients. This can be difficult to balance on a day-to-day basis because of the fast-paced nature of medicine, time constraints for clinical teaching in busy wards, and competition between students [ ]. In addition, over the past year, the experience of the global COVID-19 pandemic has vastly reduced access to bedside teaching for nursing, midwifery, and medical students [ ]. Therefore, the availability and development of VPs and immersive learning environments can play a key role in the future as an adjunct to developing clinical skills for students at all levels and at any time [ ].
The primary outcome in most studies (28/29, 97%) was knowledge. This review demonstrated that immersive technology is equally effective in knowledge gained by the student and, in some studies, reflects a higher level of knowledge retention. Regarding secondary outcomes concerning the learning experience, all studies reported overall positivity and higher satisfaction in learning, self-efficacy, and engagement with immersive technologies. Moreover, this review revealed the heterogeneity of tools and instruments used to evaluate secondary learning outcomes such as student satisfaction, self-efficacy, engagement, and learning perceptions. Several studies developed their own Likert-style assessment scales, whereas others used and adapted previously developed and validated scales for the assessment of secondary learning outcomes. According to the Association for Medical Education in Europe, “assessment tools selected should be valid, reliable, practical and have an appropriate impact on student learning” . Perhaps this highlights the need for a standardized, validated instrument to be designed specifically for immersive technology in education for the evaluation of learning outcomes related to the students’ learning experience. For example, in simulation, the National League of Nursing developed a validated standardized scale to evaluate the use of simulation and student learning experience with simulation activity, including students’ satisfaction and self-confidence, perceived learning, and engagement [ ]. Therefore, the development of a standardized instrument to evaluate learning outcomes, such as satisfaction, self-confidence, self-efficacy, and engagement, for immersive educational tools may be beneficial so that future research may be better informed by a more uniform approach to assessing learning outcomes.
Strengths and Limitations
This systematic review provides an up-to-date review on the learning outcomes of immersive technologies in university-level medical and nursing education. This study illustrates important findings on the use of immersive technologies that will provide a foundation for future research in this area. Knowledge gain with immersive technologies was shown in this review to be equal to or greater than that of traditional modalities, thus providing an evidence base for future curriculum developers and researchers alike to implement these immersive tools into university programs. A comprehensive search strategy and robust methodology support the strengths of this study. The use of the validated MERSQI tool to assess the study design of the included studies also adds to the strength of our study design.
Most of the reviewed studies developed their own bespoke immersive educational tool specific to their needs, with VR being primarily used. The favored use of VR may be due in part to the widespread availability of cost-effective 3D software and web-based material for the development of anatomy tools [, ]. However, the development of these tools has financial and resource implications, including the time spent developing content for these devices. With the increased amount of content and material, there may be an opportunity to develop a universal platform that makes content sharable and available worldwide for health care education. This may reduce the barriers to accepting and implementing this technology in health care education.
Nevertheless, we recognize that there were also some limitations to this study. Common biases exist within the methodology section, including the study eligibility criteria, identification and selection of studies, data extraction, and study appraisal. Predefined search criteria and inclusion criteria were set out in the published protocol, which aimed to reduce these biases. Within the published literature, there is heterogeneity in how VR is defined, requiring us to confine VR to include the use of a headset, providing only an immersive experience, as opposed to a 3D visualization on a computer screen. Therefore, this may have resulted in the exclusion of studies involving a 3D animation or model on a computer screen that did not place the learner in an immersive environment. Furthermore, because of the initial large database of articles retrieved, we decided to only include RCTs, as they are regarded as the highest-quality study type and also included a comparator that was a traditional learning modality. This process was conducted by 2 independent reviewers (GR and SC), and full-text inclusion was dependent on agreement by both reviewers. During the study retrieval process, bias may have occurred because of the unavailability of some studies. Authors were contacted in cases of unavailable data; however, this may have led to data availability bias and unrepresented data. This study included only preclinical university students; therefore, the value of immersive technology in the postqualification setting is unknown.
In conclusion, this systematic review demonstrates that VR, AR, and MR are beneficial educational tools in preclinical medical and nursing university education. Immersive technology is equally effective in teaching and increases learner satisfaction, self-confidence, and engagement. However, further research is required to explore the role of VR, AR, and MR in midwifery education. With the increasing availability of cost-effective immersive tools, the use of immersive technologies in health care student education is potentially very valuable.
This study was part of a thesis by GR for an MD in Medical Education, University College Dublin, Ireland. GR and SC were involved in the writing of the study protocol drafts, development of the study design, data analysis, and writing of the manuscript. AR was involved in the data analysis, writing, and review of the manuscript. MH was a study supervisor and was involved in the review of the manuscript. EM was also a study supervisor and reviewed the manuscript. FMA was a study supervisor and was involved in the study design, writing, and review of the study protocol and manuscript.
Conflicts of Interest
Comprehensive list of search terms developed for the search strategy.PDF File (Adobe PDF File), 22 KB
Validated scales used for evaluation.PDF File (Adobe PDF File), 19 KB
Medical Education Research Study Quality Instrument domain and item scores for the 29 studies.PDF File (Adobe PDF File), 32 KB
- Peterson DC, Mlynarczyk GS. Analysis of traditional versus three-dimensional augmented curriculum on anatomical learning outcome measures. Anat Sci Educ 2016;9(6):529-536 [FREE Full text] [CrossRef] [Medline]
- Jin J, Bridges SM. Educational technologies in problem-based learning in health sciences education: a systematic review. J Med Internet Res 2014;16(12):e251 [FREE Full text] [CrossRef] [Medline]
- Mangina E. 3D learning objects for augmented/virtual reality educational ecosystems. In: 2017 23rd International Conference on Virtual System & Multimedia.: IEEE; 2017 Presented at: VSMM'17; Octover 31-November 4, 2017; Dublin p. 1-6. [CrossRef]
- Bell JT, Fogler HS. The investigation and application of virtual reality as an educational tool. In: 1995 American Society of Engineering Education Annual Conference. 1995 Presented at: ASEE'95; June 25-28, 1995; Anahein.
- Salzman MC, Dede C, Loftin RB, Chen J. A model for understanding how virtual reality aids complex conceptual learning. Presence (Camb) 1999;8(3):293-316. [CrossRef]
- Ellaway R, Masters K. AMEE guide 32: e-learning in medical education part 1: learning, teaching and assessment. Med Teach 2008;30(5):455-473. [CrossRef] [Medline]
- Kapralos B, Uribe-Quevedo A, Dubrowski A. Immersive technologies for medical education. In: Lee N, editor. Encyclopedia of computer graphics and games. Ney York City: Springer International Publishing; 2018:1-8.
- Anderson EF, McLoughlin L, Liarokapis F, Peters CE, Petridis P, de Freitas S. Developing serious games for cultural heritage: a state-of-the-art review. Virtual Real 2010;14(4):255-275. [CrossRef]
- Hussein M, Nätterdal C. The benefits of virtual reality in education- a comparision study. Göteborgs Universitet. Gothenburg: University of Gothenburg; 2015. URL: http://hdl.handle.net/2077/39977 [accessed 2020-12-15]
- Pantelidis P, Chorti A, Papagiouvanni I, Paparoidamis G, Drosos C, Panagiotakopoulos T, et al. Virtual and augmented reality in medical education. In: Tsoulfas G, editor. Medical and Surgical Education-Past, Present and Future. London: IntechOpen; 2017:77-97.
- Nicola S, Stoicu-Tivadar L. Mixed reality supporting modern medical education. Stud Health Technol Inform 2018;255:242-246. [Medline]
- Maresky HS, Oikonomou A, Ali I, Ditkofsky N, Pakkal M, Ballyk B. Virtual reality and cardiac anatomy: exploring immersive three‐dimensional cardiac imaging, a pilot study in undergraduate medical anatomy education. Clin Anat 2019;32(2):238-243. [CrossRef] [Medline]
- Moro C, Štromberga Z, Raikos A, Stirling A. The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ 2017;10(6):549-559. [CrossRef] [Medline]
- Nicholson DT, Chalk C, Funnell WR, Daniel SJ. Can virtual reality improve anatomy education? A randomised controlled study of a computer-generated three-dimensional anatomical ear model. Med Educ 2006;40(11):1081-1087. [CrossRef] [Medline]
- Hansen MM. Versatile, immersive, creative and dynamic virtual 3-D healthcare learning environments: a review of the literature. J Med Internet Res 2008;10(3):e26. [CrossRef] [Medline]
- Liyanagunawardena TR, Williams SA. Massive open online courses on health and medicine: review. J Med Internet Res 2014;16(8):e191 [FREE Full text] [CrossRef] [Medline]
- Paim CP, Goldmeier S. Development of an educational game to set up surgical instruments on the mayo stand or back table: applied research in production technology. JMIR Serious Games 2017;5(1):e1 [FREE Full text] [CrossRef] [Medline]
- Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg 2012;99(10):1322-1330 [FREE Full text] [CrossRef] [Medline]
- Aksoy E. Comparing the effects on learning outcomes of tablet-based and virtual reality-based serious gaming modules for basic life support training: randomized trial. JMIR Serious Games 2019;7(2):e13442 [FREE Full text] [CrossRef] [Medline]
- Car J, Carlstedt-Duke J, Tudor Car L, Posadzki P, Whiting P, Zary N, Digital Health Education Collaboration. Digital education in health professions: the need for overarching evidence synthesis. J Med Internet Res 2019;21(2):e12913 [FREE Full text] [CrossRef] [Medline]
- Akl EA, Pretorius RW, Sackett K, Erdley WS, Bhoopathi PS, Alfarah Z, et al. The effect of educational games on medical students’ learning outcomes: a systematic review: BEME guide No 14. Med Teach 2010;32(1):16-27. [CrossRef] [Medline]
- De Gagne JC, Oh J, Kang J, Vorderstrasse AA, Johnson CM. Virtual worlds in nursing education: a synthesis of the literature. J Nurs Educ 2013;52(7):391-396. [CrossRef] [Medline]
- Tang KS, Cheng DL, Mi E, Greenberg PB. Augmented reality in medical education: a systematic review. Can Med Educ J 2020;16(11):e81-e96. [CrossRef] [Medline]
- Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ 2020;20(1):127. [CrossRef] [Medline]
- Forgione A, Guraya S. The cutting-edge training modalities and educational platforms for accredited surgical training: a systematic review. J Res Med Sci 2017;22:51. [CrossRef] [Medline]
- Stepan K, Zeiger J, Hanchuk S, Del Signore A, Shrivastava R, Govindaraj S, et al. Immersive virtual reality as a teaching tool for neuroanatomy. Int Forum Allergy Rhinol 2017;7(10):1006-1013. [CrossRef] [Medline]
- Alfalah SF, Falah JF, Muhaidat NM, Hudaib A, Koshebye D, AlHourani S. Virtual reality learning environment in embryology education. Int J Educ Pedagogical Sci 2019;13(12):1461-1464. [CrossRef]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535. [CrossRef] [Medline]
- Higgins JT, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane. 2021. URL: https://training.cochrane.org/handbook [accessed 2020-10-02]
- Cook DA, Reed DA. Appraising the quality of medical education research methods: the medical education research study quality instrument and the Newcastle-Ottawa scale-education. Acad Med 2015;90(8):1067-1076. [CrossRef] [Medline]
- Kirkpatrick DL, Kirkpatrick JD. Implementing the four levels: a practical guide for effective evaluation of training programs. Oakland: Berrett-Koehler Publishers; 2009.
- Seifert L, Manap A, Sterz J, Gerlach F, Sader R. A comparison between virtual patient and peer‑assisted learning in teaching basic medical knowledge and skills. Electron J e-Learn 2020;18(1):40-56. [CrossRef]
- Wang C, Daniel BK, Asil M, Khwaounjoo P, Cakmak YO. A randomised control trial and comparative analysis of multi-dimensional learning tools in anatomy. Sci Rep 2020;10(1):6120. [CrossRef] [Medline]
- Rossler KL, Sankaranarayanan G, Duvall A. Acquisition of fire safety knowledge and skills with virtual reality simulation. Nurse Educ 2019;44(2):88-92 [FREE Full text] [CrossRef] [Medline]
- Lombardi SA, Hicks RE, Thompson KV, Marbach-Ad G. Are all hands-on activities equally effective? Effect of using plastic models, organ dissections, and virtual dissections on student learning and perceptions. Adv Physiol Educ 2014;38(1):80-86 [FREE Full text] [CrossRef] [Medline]
- Padilha JM, Machado PP, Ribeiro A, Ramos J, Costa P. Clinical virtual simulation in nursing education: randomized controlled trial. J Med Internet Res 2019;21(3):e11529. [CrossRef] [Medline]
- Blanié A, Amorim MA, Benhamou D. Comparative value of a simulation by gaming and a traditional teaching method to improve clinical reasoning skills necessary to detect patient deterioration: a randomized study in nursing students. BMC Med Educ 2020;20(1):53 [FREE Full text] [CrossRef] [Medline]
- Liaw SY, Chan SW, Chen FG, Hooi SC, Siau C. Comparison of virtual patient simulation with mannequin-based simulation for improving clinical performances in assessing and managing clinical deterioration: randomized controlled trial. J Med Internet Res 2014;16(9):e214 [FREE Full text] [CrossRef] [Medline]
- Menzel N, Willson LH, Doolen J. Effectiveness of a poverty simulation in Second Life®: changing nursing student attitudes toward poor people. Int J Nurs Educ Scholarsh 2014;11(1):39-45. [CrossRef] [Medline]
- Gnanasegaram JJ, Leung R, Beyea JA. Evaluating the effectiveness of learning ear anatomy using holographic models. J Otolaryngol Head Neck Surg 2020;49(1):63. [CrossRef] [Medline]
- Liaw SY, Tan KK, Wu LT, Tan SC, Choo H, Yap J, et al. Finding the right blend of technologically enhanced learning environments: randomized controlled study of the effect of instructional sequences on interprofessional learning. J Med Internet Res 2019;21(5):e12537 [FREE Full text] [CrossRef] [Medline]
- Moro C, Phelps C, Redmond P, Stromberga Z. HoloLens and mobile augmented reality in medical and health science education: a randomised controlled trial. Br J Educ Technol 2021;52(2):680-694. [CrossRef]
- Hu KC, Salcedo D, Kang YN, Lin CW, Hsu CW, Cheng CY, et al. Impact of virtual reality anatomy training on ultrasound competency development: a randomized controlled trial. PLoS One 2020;15(11):e0242731 [FREE Full text] [CrossRef] [Medline]
- Engum SA, Jeffries P, Fisher L. Intravenous catheter training system: computer-based education versus traditional learning methods. Am J Surg 2003;186(1):67-74. [CrossRef] [Medline]
- Berg H, Steinsbekk A. Is individual practice in an immersive and interactive virtual reality application non-inferior to practicing with traditional equipment in learning systematic clinical observation? A randomized controlled trial. BMC Med Educ 2020;20(1):123. [CrossRef] [Medline]
- Kiesewetter J, Sailer M, Jung VM, Schönberger R, Bauer E, Zottmann JM, et al. Learning clinical reasoning: how virtual patient case format and prior knowledge interact. BMC Med Educ 2020;20(1):73 [FREE Full text] [CrossRef] [Medline]
- Schoeb DS, Schwarz J, Hein S, Schlager D, Pohlmann PF, Frankenschmidt A, et al. Mixed reality for teaching catheter placement to medical students: a randomized single-blinded, prospective trial. BMC Med Educ 2020;20(1):510 [FREE Full text] [CrossRef] [Medline]
- Noll C, von Jan U, Raap U, Albrecht UV. Mobile augmented reality as a feature for self-oriented, blended learning in medicine: randomized controlled trial. JMIR Mhealth Uhealth 2017;5(9):e139 [FREE Full text] [CrossRef] [Medline]
- Liaw SY, Ooi SW, Rusli KD, Lau TC, Tam WW, Chua WL. Nurse-physician communication team training in virtual reality versus live simulations: randomized controlled trial on team communication and teamwork attitudes. J Med Internet Res 2020;22(4):e17279 [FREE Full text] [CrossRef] [Medline]
- Ienghong K, Kotruchin P, Tangpaisarn T, Apiratwarakul K. Practical emergency ultrasound flashcards with augmented reality in teaching point-of-care ultrasound in ER. Open Access Maced J Med Sci 2021;9(E):39-42. [CrossRef]
- Sobocan M, Turk N, Dinevski D, Hojs R, Pecovnik Balon B. Problem-based learning in internal medicine: virtual patients or paper-based problems? Intern Med J 2017;47(1):99-103. [CrossRef] [Medline]
- Kockro RA, Amaxopoulou C, Killeen T, Wagner W, Reisch R, Schwandt E, et al. Stereoscopic neuroanatomy lectures using a three-dimensional virtual reality environment. Ann Anat 2015;201:91-98. [CrossRef] [Medline]
- Nickel F, Hendrie JD, Bruckner T, Kowalewski KF, Kenngott HG, Müller-Stich BP, et al. Successful learning of surgical liver anatomy in a computer-based teaching module. Int J Comput Assist Radiol Surg 2016;11(12):2295-2301. [CrossRef] [Medline]
- Berg H, Steinsbekk A. The effect of self-practicing systematic clinical observations in a multiplayer, immersive, interactive virtual reality application versus physical equipment: a randomized controlled trial. Adv Health Sci Educ Theory Pract 2021;26(2):667-682 [FREE Full text] [CrossRef] [Medline]
- Bogomolova K, van der Ham IJ, Dankbaar ME, van den Broek WW, Hovius SE, van der Hage JA, et al. The effect of stereoscopic augmented reality visualization on learning anatomy and the modifying effect of visual-spatial abilities: a double-center randomized controlled trial. Anat Sci Educ 2020;13(5):558-567. [CrossRef] [Medline]
- O’Rourke SR, Branford KR, Brooks TL, Ives LT, Nagendran A, Compton SN. The emotional and behavioral impact of delivering bad news to virtual versus real standardized patients: a pilot study. Teach Learn Med 2019;32(2):139-149. [CrossRef] [Medline]
- Du YC, Fan SC, Yang LC. The impact of multi-person virtual reality competitive learning on anatomy education: a randomized controlled study. BMC Med Educ 2020;20(1):343. [CrossRef] [Medline]
- Issleib M, Kromer A, Pinnschmidt HO, Süss-Havemann C, Kubitz JC. Virtual reality as a teaching method for resuscitation training in undergraduate first year medical students: a randomized controlled trial. Scand J Trauma Resusc Emerg Med 2021;29(1):27 [FREE Full text] [CrossRef] [Medline]
- Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA 2007;298(9):1002-1009. [CrossRef] [Medline]
- Hampton BS, Sung VW. Improving medical student knowledge of female pelvic floor dysfunction and anatomy: a randomized trial. Am J Obstet Gynecol 2010;202(6):601.e1-601.e8. [CrossRef] [Medline]
- Pears M, Yiasemidou M, Ismail MA, Veneziano D, Biyani CS. Role of immersive technologies in healthcare education during the COVID-19 epidemic. Scott Med J 2020;65(4):112-119. [CrossRef] [Medline]
- Marsh KR, Giffin BF, Lowrie Jr DJ. Medical student retention of embryonic development: impact of the dimensions added by multimedia tutorials. Anat Sci Ed 2008;1(6):252-257. [CrossRef] [Medline]
- Brewer DN, Wilson TD, Eagleson R, de Ribaupierre S. Evaluation of neuroanatomical training using a 3D visual reality model. Stud Health Technol Inform 2012;173:85-91. [Medline]
- Liesefeld HR, Fu X, Zimmer HD. Fast and careless or careful and slow? Apparent holistic processing in mental rotation is explained by speed-accuracy trade-offs. J Exp Psychol Learn Mem Cogn 2015;41(4):1140-1151. [CrossRef] [Medline]
- Persky S, McBride CM. Immersive virtual environment technology: a promising tool for future social and behavioral genomics research and practice. Health Commun 2009;24(8):677-682. [CrossRef] [Medline]
- Chen FQ, Leng YF, Ge JF, Wang DW, Li C, Chen B, et al. Effectiveness of virtual reality in nursing education: meta-analysis. J Med Internet Res 2020;22(9):e18290. [CrossRef] [Medline]
- Spencer J. Learning and teaching in the clinical environment. BMJ 2003;326(7389):591-594 [FREE Full text] [CrossRef] [Medline]
- Remtulla R. The present and future applications of technology in adapting medical education amidst the COVID-19 pandemic. JMIR Med Educ 2020;6(2):e20190 [FREE Full text] [CrossRef] [Medline]
- Shumway JM, Harden RM, Association for Medical Education in Europe. AMEE guide No. 25: the assessment of learning outcomes for the competent and reflective physician. Med Teach 2003;25(6):569-584. [CrossRef] [Medline]
|AR: augmented reality|
|MERSQI: Medical Education Research Study Quality Instrument|
|MR: mixed reality|
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses|
|PROSPERO: International Prospective Register of Systematic Reviews|
|RCT: randomized controlled trial|
|VP: virtual patient|
|VR: virtual reality|
Edited by R Kukafka; submitted 30.04.21; peer-reviewed by R Lundin, A Roy, S Mann; comments to author 28.06.21; revised version received 11.07.21; accepted 26.10.21; published 01.02.22Copyright
©Grace V Ryan, Shauna Callaghan, Anthony Rafferty, Mary F Higgins, Eleni Mangina, Fionnuala McAuliffe. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 01.02.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.