Published on 16.02.15 in Vol 17, No 2 (2015): February
Training With Virtual Patients in Transcultural Psychiatry: Do the Learners Actually Learn?
Background: The rapid increase in the number of patients with diverse ethnic backgrounds and previous exposure to severe mental trauma dictates the need for improvement in the quality of transcultural psychiatric health care through the development of relevant and effective training tools.
Objective: This study aimed to evaluate the impact of training with a virtual patient on the learner’s knowledge of posttraumatic stress disorder symptoms, clinical management, and basic communication skills.
Methods: The authors constructed an interactive educational tool based on virtual patient methodology that portrayed a refugee with severe symptoms of posttraumatic stress disorder and depression. A total of 32 resident psychiatrists tested the tool and completed a pre-interaction and post-interaction knowledge test, including skills, at the time and several weeks later.
Results: All of the participants (N=32) completed the pre-interaction and post-interaction test, and 26 (81%) of them completed the online follow-up test. The mean pre-interaction score was 7.44 (male: 7.08, female: 7.65, no statistical significance). The mean post-interaction score was 8.47, which was significantly higher (P<.001) than the pre-interaction score (mean score 7.44). The mean score for the follow-up test several weeks later was 8.38, higher than the pre-interaction score by 0.69 points but not statistically significant.
Conclusions: Our results suggest that virtual patients can successfully facilitate the acquisition of core knowledge in the field of psychiatry, in addition to developing skills such as clinical reasoning, decision making, and history taking. Repeated training sessions with virtual patients are proposed in order to achieve sustainable educational effects.
J Med Internet Res 2015;17(2):e46
The number of patients with diverse ethnic backgrounds (often having been exposed to severe mental trauma) is rapidly increasing [, ], which highlights the crucial need for effective training platforms that can provide essential knowledge and skills to care for this vulnerable group. Immigrants have higher rates of disability than host populations: social disability including unemployment [ ], mental disability such as posttraumatic stress disorder (PTSD) and major depression [ , ], as well as physical disability such as cardiovascular disease [ ] and metabolic syndrome [ ]. Unfortunately, trauma-related diagnoses are often missed in primary care [ ]. The need for new methods of acquiring proper communication skills is considered crucial, since most current systems incorporate these important skills informally in clinical training under supervision but with no direct focus on them.
Rapid technological progress during the last few decades has enabled the development of innovative educational tools, most often to supplement traditional medical curricula. Virtual patients (VPs) are broadly defined as “interactive computer simulations of real-life clinical scenarios for the purpose of medical training, education, or assessment” . They include a highly diverse group of platforms that, since the first published description in the early 1970s (10), have been developed and studied thoroughly and shown to provide a realistic, reliable, safe, and consistent learning environment for enhancing various aspects of knowledge and skills, including clinical reasoning, clinical decision making, communication, and history taking [ - ].
Very few studies have explored implementation of VPs by psychiatrists [- ], and to the best of our knowledge, no other VP system dedicated to traumatized refugees has been described, although virtual reality has recently been used and studied as a means of providing exposure therapy for veterans with combat-related PTSD [ ]. Previous papers published by our research team examined our system in terms of various aspects of user acceptance, expectations, attitudes, and educational potentials [ - ] and obtained promising results. This study examined the impact of a training session with the VP system on core knowledge related to PTSD symptomatology and clinical management, as well as basic communication skills.
Refugee Trauma Simulation System
We developed a VP system called Refugee Trauma Simulation (RT-SIM) that portrays an adult Bosnian refugee (“Mrs K.”) who presents severe symptoms of PTSD and major depression (). The VP is shown in video format by displaying appropriate prerecorded sequences depending on the questions asked by the user. The medical interview is conducted by selecting questions from a list based on suitable categories. The user can physically examine the VP, as well as order laboratory and imaging tests ( ). A preliminary assessment, including a summary of the patient’s history, differential diagnosis, and treatment plan is filled out by the learner upon completion of the virtual consultation.
An individualized, automated feedback module provided by both the VP and a virtual advisor (VA) follows. The feedback by the VP gives the patient’s perspective of the consultation, while the feedback by the VA focuses on more technical and clinical aspects of PTSD diagnostic criteria, clinical management, and basic communication skills. The feedback is designed in accordance with the learner’s actual performance and provides a brief and relevant theoretical background.
For example, if the learner did not sufficiently examine the patient’s trauma history, the VA commented: “I do not think that you sufficiently examined the patient’s trauma history by asking enough relevant questions about traumatic events. This is important in order to investigate exposure to events that could involve actual or threatened death or serious injury, or a threat to the physical integrity of self or others. If the patient reacted with intense fear, helplessness, or horror during such exposure, the first criterion for PTSD according to DSM-IV has been fulfilled”. A more detailed description of the construction of the feedback module was presented in a previously published study .
Design and Evaluation Outcomes
In response to email invitations sent by the department heads at three major university hospitals in Sweden, 32 psychiatric residents agreed to participate in our study. The participants sat in silent group rooms under supervision and were asked to interact with the VP for up to 45 minutes as they would in a real-life clinical encounter.
The pre-interaction and post-interaction outcome was a knowledge test consisting of 11 multiple-choice questions that were linked to the following fields: diagnostic criteria of PTSD according to the Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV), clinical management of patients with PTSD, and theoretical aspects of basic communication skills. The highest possible score for each test was 11 (each correct answer generating 1 point). Multiple response items, for which more than one answer was keyed as correct, were so indicated. A follow-up test, hosted by the Karolinska Institutet online learning management system (Ping-Pong), was given several weeks later. All three tests (pre-interaction, post-interaction, and follow-up) consisted of identical multiple-choice questions, though appearing in different orders. After completion of the tests, no feedback about the success rate or correct answers was provided to the participants. Explicit instructions stated that no outside help was allowed during the follow-up online test. An advisory opinion by the Stockholm Regional Ethical Review Board (2011/321-31/3) was obtained for this study, and informed consent was obtained from all of the participants.
Stata/IC 12.1 for Mac was used for statistical analysis of the data. We used descriptive statistics to analyze demographic data and the matched-pair t test in order to estimate changes in the scores between pre-interaction, post-interaction, and follow-up tests. Participants who did not complete the follow-up test were excluded from that particular analysis. P values ≤.05 were regarded as evidence of statistical significance.
All of the participants (N=32; 12 males, 20 females) completed the pre-interaction and post-interaction knowledge test. Twenty-six (81%) of the participants completed the online follow-up knowledge test (9 males, 17 females). The mean age was 35.6 (female: 35.5, men: 35.9; range 28-51), while their mean experience as psychiatric residents was 2.3 years (female: 2.4, men: 2.1; range 0-5). No statistically significant differences in age and work experience between male and female participants were found. The mean duration of the interactive session with the patient was 68 minutes (min: 25; max: 110). The mean time between the interactive session with the VP and the online follow-up test was 79 days: 78 days for men and 79 days for women (not significant, P=.78).
|Pre vs Post||-1.03||1.33||-1.51 to -0.55||-4.38||31||<.001a|
|Post vs FU||0.35||1.74||-0.36 to 1.05||1.01||25||.32|
|Pre vs FU||-0.69||2.29||-1.61 to 0.23||-1.54||25||.14|
|Pre vs Post||-0.83||1.19||-1.59 to -0.08||-2.42||11||.03a|
|Post vs FU||0.11||2.02||-1.45 to1.67||0.16||8||.87|
|Pre vs FU||-0.67||2.00||-2.20 to 0.87||-1.00||8||.35|
|Pre vs Post||-1.15||1.42||-1.82 to -0.48||-3.61||11||.002a|
|Post vs FU||0.47||1.62||-0.36 to 1.31||1.19||8||.25|
|Pre vs FU||-0.71||2.49||-1.99 to 0.58||-1.17||8||.26|
shows the mean scores on the pre-interaction, post-interaction, and follow-up knowledge test, both overall and by gender. The mean pre-interaction score was 7.44 (male: 7.08, female: 7.65, non significant), and the mean post-interaction score was 8.47, demonstrating an improvement of 1.03 points, which was highly statistically significant overall (P<.001) and by gender (male: P=.03, female: P=.002). The mean score on the follow-up knowledge test was 8.38, higher than the pre-interaction score by 0.69 points, which was not statistically significant. The follow-up score was 0.35 points lower than the post-interaction score, which was not statistically significant. Similar results were found when the data were analyzed by gender.
This paper describes a VP system that was constructed as a platform for training clinical management of traumatized refugees and was evaluated in terms of impact on the learner’s core knowledge of PTSD symptomatology and clinical management, as well as basic communication skills.
To the best of our knowledge, only one published study has previously evaluated how VPs facilitate core knowledge in the field of psychiatry in general, finding no change in knowledge of PTSD symptoms pre-intervention and post-intervention . Our results indicated a highly significant improvement on the knowledge test immediately after the training session with the VP. The fact that our participants did not receive any feedback about the success rate or correct answers after completion of the tests dramatically reduces the possibility that the improvement was due to recall rather than actual knowledge gain. The results were better on the follow-up test than the pre-interaction test but had declined and the improvement was no longer statistically significant. As far as our rather small sample allows, we can therefore conclude that a single session with the VP is not sufficient to produce a long-term impact on knowledge, which is consistent with previous findings that improvements in the performance of diagnostic tasks due to short-term training are not long-lasting [ ]. We propose research about the impact of repeated sessions over a course of several months on the durability of acquired knowledge.
It has been suggested that VPs are not an ideal methodology for facilitating core knowledge, given potential cognitive overload and the fact that less interactive methods might be more effective . Based on the system described above, we propose a VP model that combines experiential learning, as presented by Kolb [ ], through active training in a realistic and appealing virtual environment, enhanced by short theoretical frameworks integrated into automated feedback. Our results support the hypothesis that this additional element can make VP systems valuable in the acquisition of core knowledge in addition to developing skills such as clinical reasoning and decision-making.
A limitation on the generalizability of our conclusions was the rather small sample of resident psychiatrists with similar work experience and expertise. Moreover, although the pre-interaction and post-interaction tests were conducted in a strictly controlled environment with no access to outside help, the possibility that participants, despite clear instructions, obtained such help during the follow-up test in an online, non-controlled environment cannot be ruled out.
For this prospective study, we used a single-subject design, which is common in applied fields of education and psychology for which subjects serve as their own controls . This design can be effectively used to evaluate the impact of an intervention (such as virtual patients) that has not been widely studied and provide an initial clue as to its effectiveness before planning larger-scale studies with other designs and populations. However, a limitation of this methodology is its inability to know how a control group receiving another form of intervention would perform. On the other hand, a review [ ] showed that it is difficult to make a comparison with a control group that receives traditional forms of education, such as lectures, in this area since clinical reasoning demands exposure to the actual situation (as provided by clinical practice or various forms of simulation).
As a next step, we intend to conduct a randomized controlled trial that examines cognitive outcomes of using several VP cases, as well as actual patient outcomes. Future studies should also include the impact of working with the VP in pairs or groups, either at a local level or online.
The authors would like to thank Olivier Courteille, PhD, for his invaluable support in developing the RT-Sim system.
Conflicts of Interest
- UN High Commissioner for Refugees (UNHCR). UNHCR Global Trends: Displacement, The New 21st Century Challenge. 2012. URL: http://reliefweb.int/sites/reliefweb.int/files/resources/UNHCR%20GLOBAL%20TRENDS%202012_V05.pdf [accessed 2014-03-13] [WebCite Cache]
- Ekblad S, Kastrup MC. Current research in transcultural psychiatry in the Nordic countries. Transcult Psychiatry 2013 Dec;50(6):841-857. [CrossRef] [Medline]
- Hollander AC, Bruce D, Ekberg J, Burström B, Ekblad S. Hospitalisation for depressive disorder following unemployment--differentials by gender and immigrant status: a population-based cohort study in Sweden. J Epidemiol Community Health 2013 Oct;67(10):875-881. [CrossRef] [Medline]
- Sledjeski EM, Speisman B, Dierker LC. Does number of lifetime traumas explain the relationship between PTSD and chronic medical conditions? Answers from the National Comorbidity Survey-Replication (NCS-R). J Behav Med 2008 Aug;31(4):341-349 [FREE Full text] [CrossRef] [Medline]
- Mollica RF, McInnes K, Pham T, Smith Fawzi MC, Murphy E, Lin L. The dose-effect relationships between torture and psychiatric symptoms in Vietnamese ex-political detainees and a comparison group. J Nerv Ment Dis 1998 Sep;186(9):543-553. [Medline]
- Coughlin S. Post-Traumatic stress disorder and cardiovascular disease. In: Post-traumatic stress disorder and chronic health conditions. Washington DC: American Public Health Association; 2012:165-186.
- Rasmusson AM, Schnurr PP, Zukowska Z, Scioli E, Forman DE. Adaptation to extreme stress: post-traumatic stress disorder, neuropeptide Y and metabolic syndrome. Exp Biol Med (Maywood) 2010 Oct;235(10):1150-1162. [CrossRef] [Medline]
- Taubman-ben-ari O, Rabinowitz J, Feldman D, Vaturi R. Post-traumatic stress disorder in primary-care settings: prevalence and physicians' detection. Psychol Med 2001 Apr 12;31(03):555-560. [CrossRef]
- Ellaway R, Poulton T, Fors U, McGee JB, Albright S. Building a virtual patient commons. Med Teach 2008;30(2):170-174. [CrossRef] [Medline]
- Fleetwood J, Vaught W, Feldman D, Gracely E, Kassutto Z, Novack D. MedEthEx Online: a computer-based learning program in medical ethics and communication skills. Teach Learn Med 2000;12(2):96-104. [CrossRef] [Medline]
- Stevens A, Hernandez J, Johnsen K, Dickerson R, Raij A, Harrison C, et al. The use of virtual patients to teach medical students history taking and communication skills. Am J Surg 2006 Jun;191(6):806-811. [CrossRef] [Medline]
- McGee JB, Neill J, Goldman L, Casey E. Using multimedia virtual patients to enhance the clinical curriculum for medical students. Stud Health Technol Inform 1998;52 Pt 2:732-735. [Medline]
- Srinivasan M, Hwang JC, West D, Yellowlees PM. Assessment of clinical skills using simulator technologies. Acad Psychiatry 2006;30(6):505-515. [CrossRef] [Medline]
- Papadopoulos L, Pentzou AE, Louloudiadis K, Tsiatsos TK. Design and evaluation of a simulation for pediatric dentistry in virtual worlds. J Med Internet Res 2013;15(11):e240 [FREE Full text] [CrossRef] [Medline]
- Georg C, Zary N. Web-based virtual patients in nursing education: development and validation of theory-anchored design and activity models. J Med Internet Res 2014;16(4):e105 [FREE Full text] [CrossRef] [Medline]
- Lin CC, Wu WC, Liaw HT, Liu CC. Effectiveness of a virtual patient program in a psychiatry clerkship. Med Educ 2012 Nov;46(11):1111-1112. [CrossRef] [Medline]
- Kenny P, Parsons TD, Gratch J, Leuski A, Rizzo AA. Virtual patients for clinical therapist skills-training. Lecture Notes In Computer Science 2007;4722:197. [CrossRef]
- Kenny P, Parsons TD, Gratch J, Rizzo AA. Evaluation of Justina: a virtual patient with PTSD. In: Intelligent Virtual Agents: Lecture Notes In Computer Science. Berlin Heidelberg: Springer-Verlag; 2008:394-408.
- Williams K, Wryobeck J, Edinger W, McGrady A, Fors U, Zary N. Assessment of competencies by use of virtual patient technology. Acad Psychiatry 2011;35(5):328-330. [CrossRef] [Medline]
- Rizzo A, Hartholt A, Grimani M, Leeds A, Liewer M. Virtual Reality Exposure Therapy for Combat-Related Posttraumatic Stress Disorder. Computer 2014 Jul;47(7):31-37. [CrossRef]
- Ekblad S, Mollica RF, Fors U, Pantziaras I, Lavelle J. Educational potential of a virtual patient system for caring for traumatized patients in primary care. BMC Med Educ 2013;13:110 [FREE Full text] [CrossRef] [Medline]
- Pantziaras I, Courteille O, Mollica R, Fors U, Ekblad S. A pilot study of user acceptance and educational potentials of virtual patients in transcultural psychiatry. Int J Med Educ 2012 Jul 21;3:132-140. [CrossRef]
- Pantziaras I, Fors U, Ekblad S. Virtual Mrs K: The learners' expectations and attitudes towards a virtual patient system in transcultural psychiatry. J Contemp Med Edu 2014;2(2):110. [CrossRef]
- Pataki C, Pato MT, Sugar J, Rizzo AS, Parsons TD, St George C, et al. Virtual patients as novel teaching tools in psychiatry. Acad Psychiatry 2012 Sep 1;36(5):398-400. [CrossRef] [Medline]
- McGuire C, Hurley RE, Butterworth JS. Auscultatory skill: gain and retention after intensive instruction. J Med Educ 1964 Feb;39:120-131. [Medline]
- Cook DA, Triola MM. Virtual patients: a critical literature review and proposed next steps. Med Educ 2009 Apr;43(4):303-311. [CrossRef] [Medline]
- Kolb DA. Experiential learning: experience as the source of learning and development. Englewood Cliffs, NJ: Prentice-Hall; 1984.
- Cooper JO, Heron TE, Heward WL. Applied behavior analysis. Upper Saddle River, NJ: Pearson/Merrill-Prentice Hall; 2007.
|DSM-IV: Diagnostic and Statistical Manual of Mental Disorders 4th edition|
|PTSD: posttraumatic stress disorder|
|VA: virtual advisor|
|VP: virtual patient|
Edited by G Eysenbach; submitted 29.04.14; peer-reviewed by D Davies, R Ellaway; comments to author 11.09.14; revised version received 21.10.14; accepted 08.11.14; published 16.02.15
©Ioannis Pantziaras, Uno Fors, Solvig Ekblad. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 16.02.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.