This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The provision of acute medical care in rural and remote areas presents unique challenges for practitioners. Therefore, a tailored approach to training providers would prove beneficial. Although simulation-based medical education (SBME) has been shown to be effective, access to such training can be difficult and costly in rural and remote areas.
The aim of this study was to evaluate the educational efficacy of simulation-based training of an acute care procedure delivered remotely, using a portable, self-contained unit outfitted with off-the-shelf and low-cost telecommunications equipment (mobile telesimulation unit, MTU), versus the traditional face-to-face approach. A conceptual framework based on a combination of Kirkpatrick’s Learning Evaluation Model and Miller’s Clinical Assessment Framework was used.
A written procedural skills test was used to assess Miller’s learning level—
A total of 69 medical students participated in the study. Students were randomly assigned to 1 of the following 3 groups: comparison (25/69, 36%), intervention (23/69, 33%), or control (21/69, 31%). For
Our results demonstrate that simulation-based training delivered remotely, applying our MTU concept, can be an effective way to teach procedural skills. Participants trained remotely in the MTU had comparable learning outcomes (
The provision of acute care in rural and remote areas presents unique challenges. Skills related to high-acuity low-occurrence procedures and clinical encounters are particularly susceptible to degradation over time and are inadequately served through on-the-job experience alone [
SBME delivered through technologies such as telesimulation and mobile simulation has been shown to be an effective means of training medical practitioners and has helped to address some of the above constraints [
Through an iterative design process, our multidisciplinary group has developed an MTU that explores many of the challenges to the delivery of SBME to rural and remote acute care practitioners. The intention is the deployment of the MTU at a rural or remote location that could house the skills training session through communication with an off-site, skilled mentor. Such a deployment would provide trainees with the appropriate simulation equipment, a standardized training environment, and access to an experienced mentor to guide the training. To our knowledge, this is one of the few such units, which combines telecommunication and mobile simulation to deliver such training.
A rigorous, theory-based, iterative approach was followed to develop the MTU and evaluate the acceptability and feasibility of delivering training remotely using the unit. Details on the development of the MTU and training materials have been published elsewhere [
The objective of this study was to compare the educational efficacy of face-to-face versus remote delivery of educational content with respect to learner’s perceptions and objective assessment of procedural performance.
This study uses a conceptual framework based on a combination of Kirkpatrick’s Learning Evaluation Model [
Framework for Learning Assessment, based on Kirkpatrick (left) and Miller (right). Adapted from Dubrowski et al [
The base of Kirkpatrick’s model relates to subject
This study was conducted at Memorial University of Newfoundland. Training of rural and remote acute care practitioners is of particular interest in the province, as 40% of the population lives in rural areas, and the province has a relatively small population (525,000) distributed across a large geographic area (405,000 km2). Acute care is delivered at a variety of health centers and hospitals across the province. These sites are staffed by physicians, nurses, and nurse practitioners with varying levels of experience. Access to SBME opportunities is often limited. Health Research Ethics Board of Memorial University of Newfoundland approved this study.
The MTU consists of an inflatable rapid deployment tent (
The eventual goal was to deliver simulation-based training remotely through the use of a self-contained vehicle outfitted with simulation equipment necessary for delivery of a number of scenarios. However, for the purpose of our test-of-concept approach, a portable and rapid deployment tent was used.
The mobile telesimulation unit rapid deployment tent.
Overview of the setup for the mentor and the trainees in the mobile telesimulation unit.
The interior of the mobile telesimulation unit demonstrating setup for procedural training.
A randomized controlled trial design was followed. A total of 3 sessions were held to compare the learning outcomes of participants who received training remotely in the MTU versus those who received the same training face-to-face. To minimize variables affecting study outcomes, face-to-face training sessions also took place in the MTU space. A control group (ie, received no training) was included to show that the intervention group (ie, remote) was not inferior to the comparison group (ie, face-to-face), and that both instructional approaches are actually effective [
The sessions focused on teaching an important high-acuity low-occurrence procedure, chest tube insertion, using a low-fidelity setup: 3D-printed ribs, secured to a plexiglass stand, covered with low-cost simulated skin, and subcutaneous tissue (
Participants were randomly assigned to 1 of 3 groups: intervention, comparison, and control. Testing procedures were conducted before the training (pretest), after the training (posttest), and 1 week later (retention test). During the pretest, participants completed a questionnaire on demographic information, the number of times they performed or witnessed a chest tube insertion before this session, their previous experience with SBME, and their previous experience with telemedicine. Next, participants completed a written procedural skills knowledge test on a number of chest tube procedure-specific questions. The demographic questionnaire and the procedural skills knowledge test were written components used to assess whether there were differences in the baseline knowledge about the chest tube procedure within or between the groups at the start of the study. The procedural skills knowledge test was also used to measure learning after the session. This corresponds to the
To measure
After the training session, during the posttest, participants in the intervention and comparison groups were asked to evaluate their satisfaction with learning and their evaluation of the training. This corresponds to Kirkpatrick’s
Furthermore, 1 week after the training session (retention test), the participants completed a questionnaire on their experiences with the procedure in the past week. They also completed the written procedural skills knowledge test again (ie,
An emergency medicine physician with 11 years of clinical emergency room experience used the modified OSATS checklist and GRS to assess the participants’ performance on the video recordings. The reviewing physician was blinded to participants’ identity and was unaware of the phase of the study (pretest, posttest, or retention test). Overall, 12% of the videos were randomly selected for review by a second experienced emergency medicine physician. The modified OSATS checklist and GRS scores were used as the primary indicators of learning outcomes (ie,
Study design.
Setup used in the video recording of the chest tube procedure (A) and example of a completed chest tube insertion (B).
Medical students during their first and second year of training (approximately 80 students per cohort) were invited to participate in the study. Participation was voluntary and was limited by the number of slots available at a scheduled data collection time (
To measure the
Name 3 indications for chest tube placement.
Name 3 contraindications to chest tube placement.
Name 4 potential complications of chest tube placement.
Name 5 essential pieces of equipment for chest tube placement.
Participants’ performance of the chest tube procedure was evaluated using a modified OSATS checklist to measure the
This study used a modified OSATS checklist and a GRS of operative performance. The checklist consists of 10 items that are scored as done correctly or not (
Injects local anesthetic
Cuts skin with scalpel to subcutaneous tissue plane (no scything)
Uses blunt dissection to enter chest cavity
Enters pleural space above rib
Checks position with digit before inserting chest tube
Inserts chest tube safely using Kelly at the tip of the tube
Inserts correct length of chest tube into chest
Secures chest tube to chest wall with silk or nylon
Connects tube and secures to drainage system with tape
Applies airtight dressing
Global Rating Scale of operative performance.
To measure participants’ reactions to the training, participants in the remote and comparison groups were asked to evaluate the training by indicating whether they thought the MTU could play an important role in rural and remote medical training, how satisfied they were with their overall experience in the MTU, and if they would recommend the MTU approach to their colleagues. Participants were also asked to indicate their satisfaction with the learning experiences. These measures were adapted from the National League of Nursing (NLN) Student Satisfaction and Self-Confidence in Learning scales [
Participants were assigned a unique identifier, and this was used to anonymize the data before analysis with respect to their training group. Data analysis was completed using SPSS version 25. Descriptive statistics were computed for the demographic variables.
Because our data did not enable us to use the parametric repeated measures analysis of variance to analyze the pretest, posttest, and retention written procedural skills tests, we created 2 new variables (pretest minus posttest score and posttest minus retention test score). The Kruskal-Wallis test (nonparametric equivalent) was then used to compare participants’ performance on the procedural skill test between the groups.
There was acceptable interrater reliability between the 2 raters who evaluated the performance of the chest tube procedure. An excellent intraclass correlation coefficient (ICC) of 0.909 was found for the GRS, and a good ICC of 0.757 was found for the checklist. Again, limited to nonparametric techniques, we created 2 new variables: 1 variable to calculate the difference between the pre- and postchecklist and GRS scores, and the second to calculate the difference between the postchecklist and retention checklist and GRS scores. A Kruskal-Wallis test was then used to compare pretest, posttest, and retention test scores for the 3 groups (ie, intervention, comparison, and control) on the modified OSATS checklist and GRS scores.
The Mann-Whitney
For all tests, a
In total, 69 medical students participated in the study across the 3 different sessions (
The groups were very similar—mean age in low to mid-20s and relatively equally mixed between the first and second year of medical school. If there was any impact on results of students being in the first or second year of medical school, it would probably negatively influence the intervention group because a slightly higher percentage of participants in this group were in their first year. However, training on chest tube insertion is not part of the standard curriculum in the first 2 years of medical school, and most participants indicated that they had never performed or even witnessed a chest tube placement before; therefore, the presession materials and this training were the first exposures to the skill for most participants. The majority had participated in low-fidelity SBME using task trainers before, between 1 and 10 times, and the majority had never received training using telemedicine.
Participants’ experience.
Characteristics | Intervention group (n=25) | Comparison group (n=23) | Control group (n=21) | |
Age (years), mean | 25 | 23 | 21 | |
1st year | 16 (64) | 6 (26) | 9 (43) | |
2nd year | 9 (36) | 17 (74) | 12 (57) | |
Never | 24 (96) | 22 (96) | 20 (95) | |
Yes | 1 (4) | 1 (4) | 2 (5) | |
Never | 22 (88) | 20 (87) | 15 (71) | |
Yes | 3 (12) | 3 (13) | 6 (29) | |
Never | 2 (8) | 5 (22) | 4 (19) | |
1-10 times | 21 (84) | 18 (78) | 15 (71) | |
>10 times | 2 (8) | 0 (0) | 2 (9.5) | |
Never | 25 (100) | 18 (78) | 19 (91) | |
At least quarterly | 0 (0) | 5 (22) | 2 (10) |
aLow-fidelity task trainers (eg, suturing pads, airway models, and chest tube placement).
Questionnaire responses at the time of retention test (1 person from the comparison group and 2 from the control group did not complete the retention test).
Characteristics | Intervention group (n=25) | Comparison group (n=22) | Control group (n=19) | |||
No, n (%) | Yes, n (%) | No, n (%) | Yes, n (%) | No, n (%) | Yes, n (%) | |
Performed a chest tube in the past week | 23 (92) | 2 (8) | 22 (100) | 0 (0) | 19 (100) | 0 (0) |
Witnessed a chest tube in the past week | 25 (100) | 0 (0) | 22 (100) | 0 (0) | 19 (100) | 0 (0) |
Received any training or done further reading on chest tube insertions in the past week | 24 (96) | 1 (4) | 21 (96) | 1 (4) | 17 (90) | 2 (11) |
Similarly, the retention test survey, assessing exposure to chest tube insertions in the week since the training, showed no real differences between the groups. Most had not performed a chest tube since the training, witnessed a chest tube, or received any training or done any further reading on chest tube insertions (
A Kruskal-Wallis test was used to compare the results of the procedural skills knowledge test. This was a brief written test completed after receiving the presession materials but before the training session. The mean test score (out of a possible score of 15) and SD were 11.52 (2.07) for the intervention group, 10.91 (2.02) for the comparison group, and 10.76 (2.56) for the control group. There was no significant difference between the groups before starting the session (χ22=1.9;
Subsequent Kruskal-Wallis tests revealed that there were no significant differences between groups from the pretest to the posttest (χ22=4.1;
A total of 204 videos of procedural performance were included in the analysis, with 3 videos per participant (3 participants did not complete the retention test). Results of the modified OSATS checklist and GRS assessment for the 3 groups (pretraining, posttraining, and 1 week after the training) are shown in
A Kruskal-Wallis test revealed that there were statistically significant differences between the groups on the pre- and post-OSATS checklist and GRS scores (
Box plots of the modified Objective Structured Assessment of Technical Skills checklist and GRS scores. GRS: Global Rating Scale.
Satisfaction with learning and evaluation of the training measures was used to examine participants’ reaction to the training.
The results of the satisfaction with learning questions (adapted from the NLN scales) that were asked in the posttest for the intervention and comparison groups are shown in
Self-reported learning—scale of 1 (strongly disagree) to 5 (strongly agree).
Measurement item (satisfaction with learning) | Intervention group (n=25), mean (SD) | Comparison group (n=23), mean (SD) | Mann-Whitney |
||
z | |||||
The teaching methods used were helpful and effective. | 4.52 (0.71) | 4.65 (0.49) | 306.5 | 0.47 | .65 |
I enjoyed how the teacher taught the session. | 4.40 (0.82) | 4.52 (0.59) | 299.0 | 0.27 | .79 |
Participants in the intervention and comparison groups were asked to evaluate their experiences with the training session that took place physically in the MTU space. Participants indicated that the MTU could play an important role in rural medical training (4.32 and 4.48 out of 5 for the intervention and comparison groups, respectively), they were satisfied with their overall experience in the MTU (4.32 and 4.43 out of 5 for the intervention and comparison groups, respectively), and they would recommend the MTU to their colleagues for SBME (4.32 and 4.43 out of 5 for the intervention and comparison groups, respectively). A Mann-Whitney
Participants’ evaluation of training modality—scale 1 (strongly disagree) to 5 (strongly agree).
Evaluation of training modality | Intervention group (n=25), mean (SD) | Comparison group (n=23), mean (SD) | Mann-Whitney |
||
z | |||||
Do you think the MTUa could play an important role in rural medical training? | 4.32 (1.11) | 4.48 (0.51) | 276.0 | −0.27 | .79 |
How satisfied are you with your overall experience in the MTU? | 4.32 (0.56) | 4.43 (0.59) | 319.5 | 0.38 | .45 |
Would you recommend the MTU to your colleagues for simulation-based medical training? | 4.32 (0.56) | 4.43 (0.73) | 331.0 | 1.02 | .31 |
aMTU: mobile telesimulation unit.
Using a conceptual framework based on Kirkpatrick’s and Miller’s works [
Consistent with the literature, we found that subject’s knowledge level (
With respect to the
Retention tests indicated that there were no statistically significant differences in skills retention between all 3 of the groups. On average, differences between the scores on retention test–modified and posttest-modified OSATS checklist and GRS scores either stayed the same or decreased slightly for all groups. From this, we conclude that the manner of instructional delivery (either remote or face-to-face) does not impact retention.
In addition to the comparable learning outcomes, participants had similarly high levels of satisfaction with learning in the MTU. Rating the teaching methods as helpful and effective, participants indicated that, on average, they enjoyed instruction during the session. This is encouraging as satisfaction with the training, in the case of the MTU concept facilitated through a local healthcare facility, could influence commitment and readiness to transfer learning to the workplace at their own site [
Overall, participants evaluated their training experience with the MTU as positive. There were no statistically significant differences in evaluations between those who received training remotely versus those who received it face-to-face. Participants felt that the MTU could play an important role in rural medical training, they indicated that they were satisfied with their overall experience in the MTU, and they would recommend the MTU to their colleagues for SBME.
The primary limitation of this study is the relatively small sample size and the inclusion of research subjects from a single institution. However, several things help make the study more robust: (1) the inclusion of a control group; (2) the study design including pretest, posttest, and retention tests; and (3) the triangulation of the results of the modified OSATS checklist and GRS scores with 2 blinded raters demonstrating a favorable interrater reliability provides reassurance of the robustness of the study results [
There are a number of implications for future SBME and research. First, there is a shift from delivery of medical education in large urban academic centers toward distributed medical education. Technologies such as video conferencing and digital library collections have enabled this advancement and are tied to social, health, and economic benefits [
SBME is a well-established training approach, particularly for high-acuity, low-occurrence procedures and scenarios. Practitioners located in rural and remote locations particularly stand to benefit as they face a number of unique challenges with respect to simulation resources, including geographic, cost, and time constraints. This study describes an evaluation of educational efficacy comparing remote versus face-to-face mentoring for procedural skills training. To our knowledge, this study is one of a few to develop and assess SBME combining the concepts of telesimulation and mobile simulation.
We used a conceptual framework based on the combination of Kirkpatrick’s Learning Evaluation Model and Miller’s Clinical Assessment Framework to guide the study. We found that training delivered remotely through the MTU is an effective way to conduct a skills session. Those who were remotely trained had comparable learning outcomes (
Consolidated Standards of Reporting Trials flow diagram.
Differences between pre, post, and retention procedural skills knowledge tests (written).
Modified Objective Structured Assessment of Technical Skills checklist and Global Rating Scale assessment of chest tube performance, mean (standard deviation) reported.
Differences between pre, post, and retention-modified Objective Structured Assessment of Technical Skills checklist and Global Rating Scale test scores.
CONSORT‐EHEALTH checklist (V 1.6.1).
Global Rating Scale
intraclass correlation coefficient
mobile telesimulation unit
National League of Nursing
Objective Structured Assessment of Technical Skills
simulation-based medical education
This project has been supported by an Ignite grant awarded by the Research and Development Corporation of Newfoundland and Labrador. The authors thank the following organizations at the Memorial University of Newfoundland: the Tuckamore Simulation Research Collaborative for research support and advice, the Clinical Learning and Simulation Center for equipment and operational support, and Memorial University of Newfoundland MED 3D for the provision of simulation models. The authors also thank the following people for their assistance during this research project: Dr Chrystal Horwood for clinical expertise in video review; Kristopher Hoover for technical assistance and involvement in early MTU prototype development; research assistant, Megan Pollard, Samantha Noseworthy, Sarah Boyd, and Krystal Bursey; Tate Skinner (technical support); Joanne Doyle (Discipline of Emergency Medicine senior secretary); and Memorial University’s Emergency Medicine Interest Group.
None declared.