Background: Clinical reasoning is based on the declarative and procedural knowledge of workflows in clinical medicine. Educational approaches such as problem-based learning or mannequin simulators support learning of procedural knowledge. Immersive patient simulators (IPSs) go one step further as they allow an illusionary immersion into a synthetic world. Students can freely navigate an avatar through a three-dimensional environment, interact with the virtual surroundings, and treat virtual patients. By playful learning with IPS, medical workflows can be repetitively trained and internalized. As there are only a few university-driven IPS with a profound amount of medical knowledge available, we developed a university-based IPS framework. Our simulator is free to use and combines a high degree of immersion with in-depth medical content. By adding disease-specific content modules, the simulator framework can be expanded depending on the curricular demands. However, these new educational tools compete with the traditional teaching
Objective: It was our aim to develop an educational content module that teaches clinical and therapeutic workflows in surgical oncology. Furthermore, we wanted to examine how the use of this module affects student performance.
Methods: The new module was based on the declarative and procedural learning targets of the official German medical examination regulations. The module was added to our custom-made IPS named ALICE (Artificial Learning Interface for Clinical Education). ALICE was evaluated on 62 third-year students.
Results: Students showed a high degree of motivation when using the simulator as most of them had fun using it. ALICE showed positive impact on clinical reasoning as there was a significant improvement in determining the correct therapy after using the simulator. ALICE positively impacted the rise in declarative knowledge as there was improvement in answering multiple-choice questions before and after simulator use.
Conclusions: ALICE has a positive effect on knowledge gain and raises students’ motivation. It is a suitable tool for supporting clinical education in the blended learning context.
One important part of clinical education is learning and mastering medical workflows, which is the basis for mastering clinical workflows in diagnosis and therapy (clinical reasoning). Nowadays, many workflows are based on standard operation procedures (SOPs) that enable performance uniformity in the clinical daily routine . A key basis of these procedures is the learning of the underlying (declarative) knowledge, which is nowadays often supported by e-learning programs [ ]. Mastering SOPs require the ability to transfer declarative knowledge (what to do) into practical or procedural knowledge (how to do it). This is most effective in attending courses, preferably in small groups where students’ preexisting knowledge is on the same level [ ]. In current medical curricula, new educational approaches such as problem-based learning, skills lab, and mannequin simulators [ , ] play an increasing role. By repetitive training of standardized clinical settings, procedural knowledge can be internalized, which ensures a certain knowledge level before working with real patients.
The rapid development of computer technology enables implementation of new educational approaches for teaching and internalizing medical workflows in diagnosis and therapy even on home computers. Immersive patient simulators (IPSs) enable a representation of a virtual counterpart in a three-dimensional “game-like” virtual environment where students can freely interact in real time with virtual patients . Playful immersion in the digital environment provides virtual experience as students can face the consequences of different decisions (trial and error) without putting real patients at risk. Repetitive training allows internalization and consolidation of the scripts that are relevant for the necessary procedural performance [ ]. Furthermore, Web-based IPS combines immersion with the advantages of distance learning [ ].
In a thematic review, we summarized the available immersive virtual patient simulators and found that the use of such simulators in clinical education is still rare. Furthermore, there was no IPS available that combined high technical quality with an in-depth level of medical content and that was freely accessible for all students . Therefore, we developed a university-based IPS prototype called “ALICE” (Artificial Interface for Clinical Education) that is free to use, available to all interested teachers, and that enjoys high student acceptance [ ]. However, application of this new educational approach requires knowledge about the impact on knowledge gain as clinical teachers may ask whether using a simulator of this kind can really support clinical education.
We formulated the hypothesis that repetitive training with ALICE consolidates clinical reasoning and has a positive impact on knowledge gain. Therefore, we implemented a teaching module that simulates patients with a complex oncological disease. It was our aim to perform a validation of this novel educational approach and to examine whether the use of ALICE has positive impact on clinical reasoning and is a suitable tool for supporting the clinical teacher.
Immersive Virtual Patient Simulator: ALICE
Design and technical realization were previously described by our group . In brief, ALICE is a Web-based IPS that enables the student to navigate through a virtual environment from a first-person perspective similar to a video game ( ). ALICE simulates a small outpatient clinic with a treatment room where the user can interact with nonplayer characters such as nurses, patients, and other doctors ( ). The simulator starts with a short instructional case where the basic controls and functions are explained. The user is able to freely interact with the virtual patient and choose the different options (medical history, examination, diagnostic tests). The student instantly receives the desired result either as a table (eg, laboratory values), image (eg, x-ray, electrocardiogram), or video (eg, computed tomography scans, ultrasound, gastroscopy). When choosing a medically nonindicated examination, the test shows a normal finding. Once the student makes a diagnosis, he or she initiates the necessary treatments and finishes the case. The models of Miller’s pyramid [ ] and Bloom’s taxonomy were used as a scaffold for planning and assessing our simulator ( ). Level of competence was assessed on level one (factual test), level two (clinical context-based test), and partly on level three of Miller’s pyramid (performance assessment in simulated patients). ALICE helps students gain and understand knowledge and offers the opportunity to apply this knowledge on virtual patients, which are important steps in Bloom’s taxonomy in the cognitive domain [ ] ( ).
Adding Medical Content
ALICE can easily be expanded by adding content modules that include a specific disease. These teaching modules contain all information about the underlying declarative and procedural knowledge as well as patient characteristics and images. These modules can be added via a special editor. Determination of impact on clinical reasoning required the development of a new simulator module. To support or reject the hypothesis that repetitive simulator use can consolidate clinical reasoning, the following requirements were defined: The simulator should mimic a disease with a standardized workflow in diagnosis and therapy. It is mandatory that there are concurrent therapeutical concepts that depend on specific clinical findings. Moreover, this specific disease should not have been a focus in the previous clinical curriculum, to minimize the influence of already obtained knowledge.
The module was evaluated in the context of a surgical seminar held at the surgical faculty of the University of Cologne. A total of 62 third-year students were taught in a small group (<5 students), each accompanied by an experienced board-certified surgeon. Each student received the patient files from 2 real patients diagnosed with esophageal cancer. Students were not aware of the diagnosis. Students were asked to summarize the findings in a clinical reasoning summary. This included diagnosis, diagnostic tests, and suggested therapy. In addition, students answered multiple-choice questions taken from the official state examination board to determine the preexisting declarative knowledge about the underlying disease. This presimulator questionnaire was stored for later analysis. After completion of the presimulator questionnaire, students worked with ALICE. Students’ performance was reviewed immediately after each case as procedural review of students’ performance immediately after training is known to be an effective incentive for knowledge gain . A virtual instructor summarizes the performance based on a short comparison of the students’ choices and stored SOPs. Furthermore, the virtual instructor summarizes the underlying declarative knowledge and optimal procedural pattern. The debriefing ends with a video presentation of the corresponding surgical procedure and a summary of the underlying declarative knowledge ( ).
After completing the simulator cases, students had to perform a reassessment of the clinical reasoning of the 2 patients and complete a postsimulator questionnaire that contained the same questions as before. Cases were recapped and summarized after simulator evaluation by the clinical teacher in a small-group, face-to-face learning.
To test the hypothesis that simulator use has an impact on knowledge gain and students’ motivation, ALICE was validated in the form of an experimental study. The variable, simulator performance, was measured using the following parameters: correct order of the diagnostic pathway, correct or incorrect diagnosis, and correct therapy. ALICE stored the user behavior at the server level, logging students’ decisions. Students’ acceptance and their opinion about the effectiveness and applicability were determined by means of a questionnaire using a (forced choice) 6-point Likert scale (). The variable “impact on future performance (predictive validity)” is an important parameter for simulator quality [ , ]. It was tested on several levels: (1) by comparison of pre- and postsimulator clinical reasoning files, (2) by comparison of simulator performance between Cases 1 and 3, and (3) by comparison of pre- and postsimulator multiple-choice questions. Student feedback was measured using a descriptive Likert scale. Evaluation was approved as a pilot project by the Educational Committee of the Medical Faculty at the University of Cologne. The Institutional Review Board was informed and there were no objections.
Evaluation of simulator acceptance and students’ opinions were analyzed using descriptive methods such as the Likert scale. Simulator performance was analyzed using the McNemar test. Performance in the pre- and postquestionnaire multiple-choice questions was analyzed using the Student t test. P˂.05 was considered significant. Data were analyzed using the SPSS software package version 20.
The content module contains the declarative and procedural knowledge of patients diagnosed with esophageal cancer. The diagnostic and therapeutic workflow of this disease is standardized. However, this disease requires an individual risk analysis as the chosen therapy depends on the tumor stage  and functional factors [ ]. Although esophageal cancer is a rare disease in Europe, it is a common disease at our hospital. As a Center of Excellence for Surgery of the Esophagus and Stomach, we see more than 400 such patients and perform more than 160 esophagectomies annually. Because the topic of esophageal cancer was not part of the previous curriculum until that point, we were able to minimize influence of preexisting knowledge.
The esophageal cancer module was added to the ALICE framework and included 3 cases with different tumor stages and different therapeutic options (). More patients can easily be added when necessary. Declarative and procedural learning targets were based on the German medical examination regulations and served as a template for the procedural content. The correct diagnostic and treatment patterns were defined in process charts ( and ) by 2 different experienced clinical teachers and served as templates for the analysis of student performance. Clinical findings as well as corresponding images, videos, and tables from real patients were collected. Radiological images were interpreted by an experienced radiologist while gastroscopy videos were interpreted by an experienced endoscopist.
|Case 1||Case 2||Case 3|
|Diagnosis||Adenocarcinoma||Adenocarcinoma||Squamous cell carcinoma|
|Tumor stage||T3 Nx M0||T4 Nx M1||T2 Nx M0|
|Diagnostics||Standard||Plus PET computed tomography||Plus bronchoscopy|
aThe 3 cases differ in tumor stage and histology, and thus require a slightly different diagnostic pathway and therapeutical approach.
A total of 62 students participated in the survey. Students’ acceptance and their opinion about the effectiveness and applicability were determined using a (forced choice) 6-point Likert scale. Six response categories were intentionally chosen because learning performance in Germany is traditionally measured on a 6-point scale and participating students were therefore familiar with a 6-level grading system. Likert scale assessment was averaged and is summarized in. Students showed a high level of motivation when using the simulator as most of them had fun using it. Most of the students demand more e-learning in their medical curricula. These students showed a high acceptance of the simulator prototype and would frequently use a simulator of this kind. Most of the students use computers on a daily basis. The majority of the students agreed that ALICE taught new knowledge.
Predictive validity was tested by measuring the impact of simulator use on declarative knowledge. Ten multiple-choice questions that were asked before and after the simulation were correlated. In the prequestionnaire, the mean number of correct answers was 5 out of 10 (SD 1). In the postquestionnaire, the students achieved 7 out of 10 correct answers (SD 1). Hence, ALICE showed a significant impact on declarative knowledge gain (P<.01).
The impact of simulator use on future clinical decision making was tested by comparing clinical decision making before and after simulator use (). Students were asked to write down the most probable diagnosis and the suggested therapy. These answers were compared by the tutor and evaluated as “right” or “wrong.” In the prequestionnaire, 65% (40/62) of students chose the right diagnosis whereas after simulator use, 92% (57/62) of students chose the right diagnosis. In the prequestionnaire, only 32% (20/62) of students chose the right therapeutic concept, and in the postquestionnaire, 76% (47/62) of students prescribed the suitable therapy. As this improvement was significant, simulator use had a positive impact on clinical decision making.
|Parameters||Correct answers, n||P value|
|Number of multiple-choice questions (out of 10)||5 (SD 1)||7 (SD 1)||<.01|
|Students who made correct diagnosis (full text)||40||57||<.01|
|Students who choose right therapy (full text)||20||47||<.01|
|Students who made correct simulator diagnosis||48||52||Not significant|
|Students who chose correct simulator therapy||27||44||<.01|
|Students who chose correct pathway||23||21||Not significant|
Reproduction of trained content was determined by comparing the simulator performance in Case 1 with Case 3 as these cases dealt with a similar diagnosis but different tumor stagings and different therapies. The correct pathway parameter was based on the flow diagram shown in. Medically nonindicated tests as well as wrong sequence of performed tests resulted in a negative rating. Simulator use showed no positive impact on the correct pathway parameter. The diagnosis and therapy parameters were compared on students’ choices based on pull-down menus with single statements. The diagnosis parameter showed no significant improvement between the two simulator cases. However, students showed an improvement in performance for the right therapy parameter (P<.01).
This study describes the successful establishment of a simulator (ALICE) for learning decision making in oncological patients. Previous studies revealed that using virtual patient simulators can have a positive impact on learning success . Immersive patient simulators enhance virtual patient simulators thanks to an immersive environment where the user becomes part of the simulation. There is evidence that immersion plays a fundamental role in virtual reality simulators, as identification with the avatar impacts motivation and improves learning success [ ]. However, the degree of immersion is influenced by many factors and furthermore probably not all students are equally suited to this learning style [ ].
For clinical teachers, information about validity and usability are important as new educational concepts compete against the established courses. Knowledge of the SOPs in diagnosis and therapy is essential when treating oncological patients . Simulator design is based on the concept of Miller’s pyramid [ ] ( ) and was developed to support clinical education in the first three steps. Miller’s pyramid gives a vivid impression of the necessary steps in acquiring and assessing knowledge. While our simulator is designed to address levels one and two, level four (Action: Does) is assessed in vivo (eg, bedside teaching), and therefore cannot be addressed by our simulator. For level three (Shows how), the parameter on reproduction of trained content was used. However, performance on this level is usually assessed in vitro, for instance, in simulated patients in an Objective Structured Clinical Examination (OSCE) environment. It is questionable whether virtual patients can replace simulator patients in an OSCE environment. We cannot answer this question with this study as this would require a comparison of two randomly assigned groups (virtual versus OSCE patients).
However, clinical reasoning includes not only knowledge of SOPs but is also influenced by many other factors, as also summarized in Bloom’s learning taxonomy , which reveals the limitations of immersive virtual patient simulators in clinical education. In the cognitive domain, ALICE focuses on supporting foundation thinking ( ) as training of critical thinking is hard to implement within a simulation.
Moreover, two other domains of Bloom’s taxonomy are not addressed by ALICE: in the affective and psychomotor domains, weighting clinical findings and evaluating different hypotheses, professional attitude, and behavior are key in clinical education and depend on the experience of the teacher. These skills are very difficult to impart using immersive virtual patient simulators. Therefore, the gold standard on clinical education is the small-group attendance learning  accompanied by a medical teacher [ , ].
Learning in small groups is impaired by the increasing workload of hospital doctors  and restrictive working time directives. Small-group learning is most effective when students have a similar knowledge level. Here, ALICE can potentially support clinical education as the simulator can prepare students for the attendance-based courses. Students can learn the reasoning of a practitioner at their own individual pace with repetition as required. Moreover, a performance-based assessment can test the students’ problem-solving skills [ ] and ensure sufficient student preparation.
ALICE showed an impact on knowledge gain as students displayed a significant improvement in finding the right therapeutic concept after working with the simulator. This is also supported by other studies for different validity levels [, ]. As these assessments are often focused on a specific part of procedural knowledge [ ], we also measured knowledge gain in terms of declarative knowledge. The comparison of pre- and posttest performance revealed an increase in declarative knowledge.
ALICE uses a three-dimensional engine similar to computer games and enables an immersion that is known to have a positive impact on students’ motivation  and even learning performance [ ]. In this study, students were enthusiastic and motivated while learning with ALICE. This was to be expected as this concept meets the expectations of Generation Y [ ]. Nevertheless, this enthusiasm is a pitfall when it comes to analyzing student performance. Contrary to reality, students tended to deliberately select all available technical examination methods one after another as they were fascinated by the high number of images and videos. As students did not exclude nonmedically indicated tests, the comparison of simulator performance in terms of the correct pathway parameter between Cases 1 and 3 was not successful. Even when students were briefed before simulator use to select only the indicated tests, this behavior was detectable. Future simulator modification is necessary to motivate students to find the optimal way instead of clicking all available tests.
ALICE is an appropriate educational tool for teaching procedural knowledge. It has a positive effect on knowledge gain and boosts student motivation. When used as required preparation in education, it can possibly lead to more efficient bedside teaching.
Conflicts of Interest
- Papakonstantinou D, Poulymenopoulou M, Malamateniou F, Vassilacopoulos G. Enabling the use of enhanced medical SOPs by an mLearning training solution. Stud Health Technol Inform 2013;190:86-88. [Medline]
- Back DA, Haberstroh N, Sostmann K, Schmidmaier G, Putzier M, Perka C, et al. High efficacy and students' satisfaction after voluntary vs mandatory use of an e-learning program in traumatology and orthopedics—A follow-up study. J Surg Educ 2014;71(3):353-359. [CrossRef] [Medline]
- Sutkin G, Wagner E, Harris I, Schiffer R. What makes a good clinical teacher in medicine? A review of the literature. Acad Med 2008 May;83(5):452-466. [CrossRef] [Medline]
- Zendejas B, Brydges R, Wang AT, Cook DA. Patient outcomes in simulation-based medical education: A systematic review. J Gen Intern Med 2013 Aug;28(8):1078-1089 [FREE Full text] [CrossRef] [Medline]
- Gerdes B, Hassan I, Maschuw K, Schlosser K, Bartholomäus J, Neubert T, et al. Instituting a surgical skills lab at a training hospital [Article in German]. Chirurg 2006 Nov;77(11):1033-1039. [CrossRef] [Medline]
- Dev P, Heinrichs WL, Youngblood P. CliniSpace: A multiperson 3D online immersive training environment accessible through a browser. Stud Health Technol Inform 2011;163:173-179. [Medline]
- Gutiérrez F, Pierce J, Vergara VM, Coulter R, Saland L, Caudell TP, et al. The effect of degree of immersion upon learning performance in virtual reality simulations for medical education. Stud Health Technol Inform 2007;125:155-160. [Medline]
- Jayakumar N, Brunckhorst O, Dasgupta P, Khan MS, Ahmed K. e-Learning in surgical education: A systematic review. J Surg Educ 2015 Jun 22:1-13 (forthcoming). [CrossRef] [Medline]
- Kleinert R, Wahba R, Chang D, Plum P, Hölscher AH, Stippel DL. 3D immersive patient simulators and their impact on learning success: A thematic review. J Med Internet Res 2015;17(4):e91 [FREE Full text] [CrossRef] [Medline]
- Kleinert R, Heiermann N, Wahba R, Chang DH, Hölscher AH, Stippel DL. Design, realization, and first validation of an immersive web-based virtual patient simulator for training clinical decisions in surgery. J Surg Educ 2015 Jun 18:1. [CrossRef] [Medline]
- Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990 Sep;65(9 Suppl):S63-S67. [Medline]
- Porter RD, Schick IC. Revisiting Bloom's taxonomy for ethics and other educational domains. J Health Adm Educ 2003;20(3):167-188. [Medline]
- Welke TM, LeBlanc VR, Savoldelli GL, Joo HS, Chandra DB, Crabtree NA, et al. Personalized oral debriefing versus standardized multimedia instruction after patient crisis simulation. Anesth Analg 2009 Jul;109(1):183-189. [CrossRef] [Medline]
- Funke K, Bonrath E, Mardin WA, Becker JC, Haier J, Senninger N, et al. Blended learning in surgery using the Inmedea Simulator. Langenbecks Arch Surg 2013 Feb;398(2):335-340. [CrossRef] [Medline]
- Knight JF, Carley S, Tregunna B, Jarvis S, Smithies R, de FS, et al. Serious gaming technology in major incident triage training: A pragmatic controlled trial. Resuscitation 2010 Sep;81(9):1175-1179. [CrossRef] [Medline]
- Rubenstein J, Shaheen NJ. Epidemiology, diagnosis, and management of esophageal adenocarcinoma. Gastroenterology 2015 Aug;149(2):302-317.e1. [CrossRef] [Medline]
- Schneider PM, Vallböhmer D, Brabender J, Hölscher AH. Preoperative evaluation of prognostic factors in esophageal squamous cell cancer [in German]. Chirurg 2005 Nov;76(11):1011-1017. [CrossRef] [Medline]
- Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: A systematic review and meta-analysis. Acad Med 2010 Oct;85(10):1589-1602. [CrossRef] [Medline]
- Kuhlen TW, Hentschel B. Quo vadis CAVE: does immersive visualization still matter? IEEE Comput Graph Appl 2014;34(5):14-21. [CrossRef] [Medline]
- Wöckel A, Kurzeder C, Geyer V, Novasphenny I, Wolters R, Wischnewsky M, et al. Effects of guideline adherence in primary breast cancer—A 5-year multi-center cohort study of 3976 patients. Breast 2010 Apr;19(2):120-127. [CrossRef] [Medline]
- Degen C, Weigl M, Glaser J, Li J, Angerer P. The impact of training and working conditions on junior doctors' intention to leave clinical practice. BMC Med Educ 2014;14:119 [FREE Full text] [CrossRef] [Medline]
- Barrows HS, Williams RG, Moy RH. A comprehensive performance-based assessment of fourth-year students' clinical skills. J Med Educ 1987 Oct;62(10):805-809. [Medline]
- Khan H, Taqui AM, Khawaja MR, Fatmi Z. Problem-based versus conventional curricula: Influence on knowledge and attitudes of medical students towards health research. PLoS One 2007;2(7):e632 [FREE Full text] [CrossRef] [Medline]
- Stetzik L, Deeter A, Parker J, Yukech C. Puzzle-based versus traditional lecture: Comparing the effects of pedagogy on academic performance in an undergraduate human anatomy and physiology II lab. BMC Med Educ 2015;15:107 [FREE Full text] [CrossRef] [Medline]
- Huerta R. Measuring the Impact of Narrative on Player's Presence and Immersion in a First Person Game Environment [master's disseration]. Ann Arbor, MI: The University of Texas - Pan American; 2012:1-78.
- Schlitzkus LL, Schenarts KD, Schenarts PJ. Is your residency program ready for Generation Y? J Surg Educ 2010;67(2):108-111. [CrossRef] [Medline]
|ALICE: Artificial Interface for Clinical Education|
|IPS: immersive patient simulator|
|OSCE: Objective Structured Clinical Examination|
|SOP: standard operating procedure|
Edited by G Eysenbach; submitted 13.08.15; peer-reviewed by E Zhu, D Davies; comments to author 03.09.15; revised version received 02.10.15; accepted 03.11.15; published 17.11.15Copyright
©Robert Kleinert, Nadine Heiermann, Patrick Sven Plum, Roger Wahba, De-Hua Chang, Martin Maus, Seung-Hun Chon, Arnulf H Hoelscher, Dirk Ludger Stippel. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.11.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.