This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Simulation in virtual environments has become a new paradigm for surgeon training in minimally invasive surgery (MIS). However, this technology is expensive and difficult to access.
This study aims first to describe the development of a new gesture-based simulator for learning skills in MIS and, second, to establish its fidelity to the criterion and sources of content-related validity evidence.
For the development of the gesture-mediated simulator for MIS using virtual reality (SIMISGEST-VR), a design-based research (DBR) paradigm was adopted. For the second objective, 30 participants completed a questionnaire, with responses scored on a 5-point Likert scale. A literature review on the validity of the MIS training-VR (MIST-VR) was conducted. The study of fidelity to the criterion was rated using a 10-item questionnaire, while the sources of content-related validity evidence were assessed using 10 questions about the simulator training capacity and 6 questions about MIS tasks, and an iterative process of instrument pilot testing was performed.
A
The development of gesture-based 3D virtual environments for training and learning basic psychomotor skills in MIS opens up a new approach to low-cost, portable simulation that allows ubiquitous learning and preoperative warm-up. Fidelity to the criterion was duly evaluated, which allowed a good enough prototype to be achieved. Content-related validity evidence for SIMISGEST-VR was also obtained.
The emergence of minimally invasive surgery (MIS) in the mid-1980s [
Simulators for skill learning in MIS can be classified into 3 large groups: (1) traditional box trainers, (2) augmented reality simulators (hybrids), and (3) virtual reality (VR) simulators [
For the development of the simulator used in this study, the researchers adopted the design-based research (DBR) paradigm, also known as
The first aim of this study was to describe the development of a web-based 3D VR simulator mediated by a gesture interface device (LMC) for learning basic psychomotor skills in MIS, called gesture-mediated simulator for MIS–VR (SIMISGEST-VR). The device is characterized by its portability and low cost, as well as the possibility of learning and training at any time and place (ubiquitous learning). The second aim of this study was to evaluate fidelity to the criterion and to find sources of content-related validity evidence for SIMISGEST-VR.
This is a descriptive report of the development, using a DBR paradigm, of a gesture-mediated simulator for learning basic psychomotor skills and of the prospective evaluation of the data obtained from Likert scale surveys to evaluate fidelity to the criterion and the sources of content-related evidence. To this end, the study participants rated fidelity to the criterion using a 10-item questionnaire about its ease of use, relevance as a tool for simulation in MIS, degree of correspondence between the movements of the forceps and their representation in the virtual space, and feedback. The sources of content-related validity evidence were (1) a literature review on a previously validated tool, the MIST-VR, and (2) an expert panel that answered 10 questions about the training capacity and 6 questions about each proposed task, with responses scored on a 5-point Likert scale that rated the extent to which the test content represented the domain evaluated. An iterative process of simulator development was performed using pilot testing by surgeons, engineers, and education experts until a
The hypotheses were as follows:
It is possible to develop a portable, low-cost, gesture-mediated simulator using the LMC for training and learning basic psychomotor skills in MIS.
The 3D virtual environment and the proposed tasks showed fidelity to the criterion.
It is possible to demonstrate sources of evidence for the content validity of the test items.
The first step of the validation process was to define the construct and proposed interpretation. In this study, the general construct is psychomotor skills in surgery, specifically basic psychomotor skills in MIS. The assumptions and proposed interpretations are that the 3D virtual environment is faithful to the criterion and the tasks adapted from the MIST-VR represent the construct that is intended to be measured. The instrument under investigation is a contactless, gesture-mediated simulator that uses the LMC (construct context). To determine the current use of gesture-mediated interfaces in surgery, especially in the field of surgical simulation, a systematic literature review was conducted [
To develop a new type of web-based 3D VR simulator mediated by a gesture interface device (LMC) for learning basic psychomotor skills in MIS, a group consisting of a pediatric surgeon, systems engineer, industrial designer, and specialists in education was formed. The following technical elements were assembled: an electronic device (LMC), a computer program for the development of the 3D environment, a computer, hardware devices with no electronic components, and a database administrator.
In May 2012, a sensor was launched based on the principle of infrared optical tracking, which detects the positions of fine objects such as fingertips or pen tips in a Cartesian plane. Its interaction zone is an inverted cone of approximately 0.23 m³, and it has a motion detection range that fluctuates between 20 mm and 600 mm [
The LMC has been used as a tool to manipulate medical images in the fields of interventional radiology and image-guided surgery or when there is a risk of contamination through contact (eg, autopsy rooms). It has also been used for touchless control of operating lights and tables and simulation in MIS and robotic surgery using physical or VR simulators [
The 3D virtual environment with MIS tasks was created using a tool for developing games, Unity3D, which allows apps to be developed that are independent of the operating system or device [
The basis for the development of this environment was the MIST-VR, presented in 1997. This device is a low-cost, nonprocedural simulator that provides a large variety of metric data for analysis [
The basic psychomotor skills in MIS that can be learned using the MIST-VR are navigation-coordination, touching, grasping, stretching-traction, translocation, and electrocautery [
The computer displays the 3D virtual environment, records the metrics, stores them on a database, and provides feedback using graphs that show the score obtained after each exercise. The virtual environment developed runs on both PC and iOS operating systems.
The mechanical devices are represented by 2 MIS forceps that do not need to be functional, 2 support devices for the forceps with an entry trocar simulator, 1 support device for the LMC, and 1 pad for mounting the support devices.
During the development of the virtual environment, the types of specificity recommended by Bowman et al [
The study was performed over a period of 3 months at different locations: XXXIV Brazilian Congress of Paediatric Surgery (Campo Grande, Brazil); Hospital Vall d’Hebron (Barcelona, Spain); and Hospital Infantil de la Cruz Roja (Manizales, Colombia). A total of 22 experienced surgeons (performed more than 100 MIS procedures) and 8 pediatric and general surgery residents (referent group, performed less than 100 MIS procedures) assisted in an informative session on the characteristics of the project, watched a demonstration video of the different tasks supported by the simulator, and had 2 opportunities to perform each of the tasks on the simulator. The performance metrics were not taken into account during this study, as the emphasis was placed on the assessment of the tool by those surveyed.
The first source of content validity for the SIMISGEST-VR sought to identify the main sources of validity evidence for the MIST-VR, as well as the studies that have demonstrated such validity.
First, a demographic survey was administered that included questions on the level of training as a surgeon and level of experience in MIS, as well as experience with video games. The different factors in the evaluation of fidelity to the criterion and content validity study were assessed using a Likert scale, where 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, and 5=strongly agree [
The questionnaire to assess fidelity to the criterion evaluated 10 aspects, while the content validity rated the training capacity and the tasks. In terms of the training capacity, 6 aspects were evaluated, and each of the 6 tasks (
This study used SIMISGEST-VR with 6 tasks and their respective metrics and feedback. The hardware and software components of the simulator are described in phase 1: Development of SIMISGEST-VR of this paper.
Normality was tested using the Shapiro-Wilk test. The distribution of the variables was not normal. The Likert scale median and interquartile range differences between the levels of education and experience were compared using the Kruskal-Wallis test. A statistically significant level <0.05 was established. The analysis was performed using Stata version 15.0 (StataCorp).
The virtual environment consists of the following modules:
Except for Task 3, all tasks have the option of configuring the dominant hand during the exercise. Task 3 requires the simultaneous use of both hands and therefore both play a dominant function.
The web-based virtual environment runs on PC and iOS platforms.
These exercises are based on the instructional strategy known as
Description of the tasks and their surgical equivalents.
Taska | Description | Surgical equivalent |
Task 1: Grip and placement | Take the sphere with one hand and move it to a new location within the workspace | Gripping and retraction of a tissue to a given position, placement of clips and hemostasis, and use of extractor bags |
Task 2: Transfer and placement of an object | Take the sphere, transfer it to another instrument, and place it inside a hollow cylinder | Transfer of a needle between a clamp and a needle holder |
Task 3: Cross | Instruments travel along a surface in a 3D cylinder | Small intestine exploration |
Task 4: Removal and reinsertion of instruments | Removal of the instruments from the operative site and reinsertion | One instrument stabilizes one organ while the other is removed from the field and reintroduced |
Task 5: Diathermy | Cauterize a series of targets located in a fixed sphere | Cauterize a bleeding blood vessel |
Task 6: Target manipulation and diathermy | Take the sphere with the instrument and place it inside a virtual space represented by a cube. Cauterize a series of targets with the other hand | Present and set a target to cauterize |
aAdapted from [
Immediate feedback.
Performance history and terminal feedback curve.
The metrics were established using 5 parameters:
The haptic sensation and the concurrent feedback are simulated using sound signals, color changes in the objects, and movement of the object when an undue collision occurs between the different components of the environment or when an error occurs during the exercise. At the end of each task, the system provides information on the presence or absence of errors, the efficacy and efficiency, and the time required (immediate feedback). At the end of each training session, the system provides a series of graphs and tables that show the performance over time; this is the terminal feedback (
The data generated by the program were initially stored on an independent Structured Query Language database engine. However, during the development, this database was integrated into the virtual environment, which facilitated the acquisition of the users’ demographic data, registration of all the data provided by the metrics, and generation of reports of the users’ demographic and performance data. This information is stored on the computer on which the tests are performed.
Two laparoscopic forceps were used. These MIS forceps did not need to be functional.
In the initial phase of development, the researchers used a prototype that did not have support devices (
The final artifact with all its components assembled is shown in
In
The changes shown in
Initial version of the prototype without support devices for the forceps.
The final version of the simulator once the nonelectronic hardware devices had been added: the pad and support devices for the forceps and the Leap Motion Controller.
Diagram of the artefact.
Initial attempts at interaction between minimally invasive surgery forceps and Leap Motion Controller within a basic 3D virtual environment.
The first functional version of the virtual environment before the feedback given by surgeons with expertise in minimally invasive surgery.
Good enough prototype of the web-based 3D virtual environment: Task 1.
Process of obtaining the good enough prototype.
Element | Initial prototype | Problem | Functional prototype | Output |
MISa forceps | The shaft of the forceps is black | Difficulties in the detection of the forceps by the LMCb | The shaft of the forceps is white | Notable improvement in detection of the forceps by the LMC |
Support devices | No support devices | Fulcrum effect not reproduced | Design of support devices | Reproduction of fulcrum effect |
Mounting pad | No mounting pad | The hardware pieces (LMC and support devices) are independent, and there is no standard arrangement | Standardized integration of the pieces in the mounting pad | Physical stability of the model |
Position of the LMC | Completely horizontal, 90 degree with regard to the screen | Difficulties in the detection of the forceps by the LMC | A forward 45-degree angle was applied with regard to the screen | Interference between the forceps when detected by the LMC was eliminated |
First prototype of the 3D virtual environment ( |
Tests on the interaction between the forceps and the objects in the virtual environment | Difficulty for interaction between the forceps and the objects in the environment | Trial and error tests on interaction by modifying LMC and instrumental variables | Complete interaction achieved |
Second prototype of the 3D virtual environment ( |
Functional environment in the 6 tasks | Quadrangular shapes in the environment | Circular shapes in the |
An abstract 3D virtual environment with circular shapes |
SQLc database engine not integrated into the simulation software | A software program should be installed in addition to the simulation program | Redesign of the model and data capture and storage | Feedback and metrics complete and integrated into the SQLite |
aMIS: minimally invasive surgery.
bLMC: Leap Motion Controller.
cSQL: Structured Query Language.
The next step in the process was the evaluation of fidelity to the criterion and the process of subjective content validity. The results are described below.
A total of 30 people with an average age of 42 years (SD 2.2) participated in the study; 53% (n=16/30) were men. Those surveyed came from Colombia (n=14), Spain (n=8), Argentina (n=3), Brazil (n=2), Uruguay (n=2), and France (n=1).
In terms of the use of video games, most (n=22/30, 73%) of those surveyed had no experience with these app; 62% (n=5/8) of those who used video games were women. Of those with experience in video games (n=8), only 1 played them weekly, while the rest played them once a month (n=3) or occasionally (n=4). The mean age of those with no experience in video games was 44 years (SD 2.7), compared with 37 years (SD 3.5) for those with experience (
Only 33% (n=10/30) of the participants had experience with VR devices, and only one-third used them occasionally.
Most of the surveyed participants had previous experience with simulators. In terms of the level of operating experience, 54% (n=14/26) of the respondents with experience with simulators had an intermediate or advanced operating level, followed by those with a basic operating level (n=10/26, 38%). Among participants who had experience with simulators [
The demographic profile questionnaire can be found in
Demographic profile according to the level of experience and training (N=30).
Demographic variable | Level of experience | Level of training | ||||||
|
Basic manipulation (n=3)a | Basic operating level (n=11)b | Intermediate operating level (n=8)c | Advanced operating level (n=8)d | Practicing surgeon (n=21) | Resident (n=8) | Other (n=1)e | |
Age (years), mean (SD) | 26 (0.6) | 40 (4.3) | 43 (3.3) | 49 (2.9) | 47 (2.2) | 27 (0.6) | 49 (—f) | |
|
||||||||
|
Male | 0 | 3 | 5 | 8 | 15 | 0 | 1 |
|
Female | 3 | 8 | 3 | 0 | 6 | 8 | 0 |
|
||||||||
|
Yes | 1 | 1 | 4 | 2 | 6 | 1 | 1 |
|
No | 2 | 10 | 4 | 6 | 16 | 6 | 0 |
|
||||||||
|
Yes | 2 | 10 | 6 | 8 | 19 | 7 | 0 |
|
No | 1 | 1 | 2 | 0 | 3 | 0 | 1 |
|
||||||||
|
Physical | 2 | 6 | 3 | 5 | 12 | 3 | 1 |
|
Hybrid and augmented reality | 0 | 2 | 2 | 2 | 4 | 2 | 0 |
|
Virtual reality | 0 | 2 | 1 | 1 | 2 | 2 | 0 |
|
No experience | 1 | 1 | 2 | 0 | 3 | 1 | 0 |
aBasic manipulation of the camera and/or retraction with forceps.
bBasic operating level (cholecystectomy and appendectomy).
cIntermediate operating level (fundoplication).
dAdvanced operating level.
eOther: an engineer highly experienced in the design of instruments and devices for minimally invasive surgery simulation.
fNot available because there was only one observation.
gMIS: minimally invasive surgery.
In terms of the fidelity to the criterion, none of the respondents strongly disagreed with any of the items asked. The rating of
In terms of ease of use, 73% (n=22/30) and 27% (n=8/30) assigned a rating of 5 and 4, respectively. The same results were obtained when the navigation menu was assessed. With regard to the relevance of the tool as a simulator, 73% (n=22/30) assigned a score of 5 and 20% (n=6/30) assigned a score of 4.
When assessing the capacity of the physical devices to simulate the fulcrum effect, 73% (22/30) assigned a score between 4 and 5, 17% (n=5/30) assigned a score of 3, and 10% (n=3/30) assigned a score of 2. For this last rating, in terms of the level of training, 2 were practicing surgeons and 1 was a resident, whereas in terms of the level of experience, one corresponded to basic manipulation, one to intermediate operating level, and another to advanced level.
In terms of how the movements of the forceps were represented in the virtual environment, 73% (n=22/30) rated this as 4 or 5, 23% (n=7/30) assigned a score of 3, and only one of the participants (n=9/30, 3%) assigned a score of 2 (level of training=practicing surgeon and level of experience=intermediate).
When assessing how appropriately the tool simulates the movements of MIS, 83% (n=25/30) rated the question as 4 or 5. All respondents (n=30/30, 100%) rated the design as attractive, with scores of 4 or 5. Almost all surveyed respondents (n=29/30, 97%) assigned ratings of 4 or 5 to the innovation factor, the capacity to provide feedback, and to the question of whether the latter was adequate.
The fidelity to the criterion study questions can be found in
Fidelity to the criterion and content validity according to the level of training.
Variable | Resident (n=8) | Practicing surgeon (n=21) | Othera (n=1) | ||||||||||||||
|
Median | IQR | Median | IQR | Median | IQR |
|
||||||||||
|
|||||||||||||||||
|
Ease of use | 5 | 4.5-5 | 5 | 4-5 | 5 | 5-5 | .88 | |||||||||
|
Navigation menu | 5 | 5-5 | 5 | 4-5 | 5 | 5-5 | .62 | |||||||||
|
Relevance as a learning tool | 5 | 4-5 | 5 | 5-5 | 5 | 5-5 | .73 | |||||||||
|
Fulcrum effect | 3.5 | 3-4 | 5 | 4-5 | 4 | 4-4 | .13 | |||||||||
|
Representation of the physical forceps in the virtual environment | 4 | 3-5 | 4 | 4-5 | 5 | 5-5 | .56 | |||||||||
|
Simulation of the movements in MISc | 4 | 4-4 | 4 | 4 - 5 | 5 | 5-5 | .18 | |||||||||
|
Innovation | 5 | 4.5-5 | 5 | 5-5 | 5 | 5-5 | .90 | |||||||||
|
Graphic design | 4.5 | 4-5 | 5 | 4-5 | 5 | 5-5 | .69 | |||||||||
|
Feedback | 5 | 4-5 | 5 | 5-5 | 5 | 5-5 | .79 | |||||||||
|
Relevance of the feedback | 4 | 4-5 | 5 | 4-5 | 5 | 5-5 | .43 | |||||||||
|
|||||||||||||||||
|
Hand-eye coordination | 4.5 | 4-5 | 5 | 4-5 | 5 | 5-5 | .66 | |||||||||
|
Depth perception | 4 | 3.5-5 | 5 | 4-5 | 5 | 5-5 | .41 | |||||||||
|
Basic psychomotor skills learning | 4.5 | 4-5 | 5 | 5-5 | 5 | 5-5 | .42 | |||||||||
|
Basic steps of MIS | 4 | 4-5 | 5 | 4-5 | 5 | 5-5 | .64 | |||||||||
|
Metrics | 4 | 3-5 | 4 | 4-5 | 5 | 5-5 | .43 | |||||||||
|
Ubiquitous learning | 4 | 4-5 | 5 | 4-5 | 5 | 5-5 | .31 | |||||||||
|
|||||||||||||||||
|
Task 1 | 3.5 | 3-4 | 4 | 4-5 | 5 | 5-5 | .19 | |||||||||
|
Task 2 | 4 | 4-4.5 | 4 | 3-5 | 5 | 5-5 | .41 | |||||||||
|
Task 3 | 4 | 3.5-4 | 4 | 3-5 | 5 | 5-5 | .40 | |||||||||
|
Task 4 | 4 | 3-5 | 5 | 4-5 | 2 | 2-2 | .21 | |||||||||
|
Task 5 | 4.5 | 4-5 | 5 | 4-5 | 5 | 5-5 | .65 | |||||||||
|
Task 6 | 4 | 4-4 | 5 | 4-5 | 5 | 5-5 | .02 |
aOther: An engineer highly experienced in the design of instruments and devices for minimally invasive surgery simulation.
bFor fidelity to the criterion questions, see
cMIS: minimally invasive surgery.
dFor content validity questions, see
eFor task descriptions, see
Fidelity to the criterion and content validity according to the level of experience.
Variable | Basic manipulation (n=3) | Basic operating level (n=11) | Intermediate operating level (n=8) | Advanced operating level (n=8) | ||||||||||||||||
|
Median | IQR | Median | IQR | Median | IQR | Median | IQR |
|
|||||||||||
|
||||||||||||||||||||
|
Ease of use | 5 | 4-5 | 5 | 4-5 | 5 | 4.5-5 | 4.5 | 4-5 | .84 | ||||||||||
|
Navigation menu | 5 | 4-5 | 5 | 5-5 | 5 | 4.5-5 | 4.5 | 4-5 | .51 | ||||||||||
|
Relevance as a learning tool | 5 | 2-5 | 5 | 4-5 | 5 | 5-5 | 5 | 4-5 | .83 | ||||||||||
|
Fulcrum effect | 4 | 2-5 | 4 | 3-5 | 5 | 4-5 | 4 | 4-5 | .66 | ||||||||||
|
Representation of the physical forceps in the virtual environment | 5 | 3-5 | 4 | 4-5 | 4 | 3.5-5 | 4.5 | 3-5 | .96 | ||||||||||
|
Simulation of the movements in MISb | 4 | 4-4 | 4 | 4-5 | 4.5 | 3.5-5 | 4 | 3.5-4.5 | .70 | ||||||||||
|
Innovation | 5 | 3-5 | 5 | 5-5 | 5 | 4.5-5 | 5 | 4.5-5 | .95 | ||||||||||
|
Graphic design | 4 | 4-5 | 5 | 4-5 | 5 | 4.5-5 | 4 | 4-5 | .41 | ||||||||||
|
Feedback | 5 | 4-5 | 5 | 4-5 | 5 | 5-5 | 4.5 | 4-5 | .42 | ||||||||||
|
Relevance of the feedback | 4 | 4-5 | 5 | 4-5 | 5 | 4.5-5 | 4.5 | 4-5 | .66 | ||||||||||
|
||||||||||||||||||||
|
Hand-eye coordination | 4 | 4-5 | 5 | 4-5 | 5 | 5-5 | 4.5 | 4-5 | .77 | ||||||||||
|
Depth perception | 5 | 4-5 | 5 | 4-5 | 5 | 4-5 | 4.5 | 4-5 | .95 | ||||||||||
|
Basic psychomotor skills learning | 4 | 3-5 | 5 | 4-5 | 5 | 5-5 | 5 | 4-5 | .45 | ||||||||||
|
Basic steps of MIS | 4 | 4-5 | 5 | 4-5 | 5 | 4.5-5 | 4 | 4-4.5 | .33 | ||||||||||
|
Metrics | 4 | 3-5 | 4 | 3-5 | 4.5 | 4-5 | 4.5 | 4-5 | .75 | ||||||||||
|
Ubiquitous learning | 4 | 4-5 | 5 | 4-5 | 5 | 5-5 | 4.5 | 4-5 | .46 | ||||||||||
|
||||||||||||||||||||
|
Task 1 | 3 | 3-5 | 4 | 3-5 | 4 | 3.5-5 | 4 | 3.5-4.5 | .88 | ||||||||||
|
Task 2 | 4 | 4-5 | 4 | 4-5 | 4 | 3.5-5 | 3.5 | 2-5 | .76 | ||||||||||
|
Task 3 | 4 | 2-5 | 4 | 3-4 | 4.5 | 3.5-5 | 4 | 3.5 – 5 | .76 | ||||||||||
|
Task 4 | 2 | 2-5 | 4 | 4-5 | 5 | 5-5 | 4 | 3.5-5 | .18 | ||||||||||
|
Task 5 | 4 | 4-5 | 5 | 4-5 | 5 | 4.5-5 | 4.5 | 4-5 | .70 | ||||||||||
|
Task 6 | 4 | 4-4 | 4 | 4-5 | 5 | 5-5 | 4.5 | 4-5 | .12 |
aFor fidelity to the criterion questions, see
bMIS: minimally invasive surgery.
cFor content validity questions, see
dFor task descriptions, see
With regard to content validity, none of the items evaluated for the training capacity were rated as 1, although, in the case of hand-eye coordination by a practicing surgeon with an advanced operating level and the depth perception by a practicing surgeon with an intermediate operating level, the hand-eye coordination and depth perception were rated as 2. Almost all of those surveyed (n=28/30, 93%) rated the hand-eye coordination as 4 or 5, while 87% (n=26/30) gave this score for depth perception.
The highest-rated item was the one that considered that the prototype could be a solution for ubiquitous learning in MIS: 100% (n=30/30) of those surveyed rated it as 4 or 5. With regard to the evaluation of the metrics, 17% (n=5/30) of those surveyed rated them as 3, while the remaining participants (n=25/30) rated them as 4 or 5.
Almost all respondents (n=29/30, 97%) considered that the SIMISGEST-VR enables learning of basic psychomotor skills in MIS, with ratings of 4 and 5; whereas, 93% (n=28/30) agreed that the tasks reflect the basic steps of a minimally invasive procedure, with ratings of 4 and 5.
An analysis of the evaluation of the tasks, in general, showed that the following were rated between 4 and 5: Task 1 received this rating from 70% (n=21/30) of those interviewed; Task 2 from 77% (n=23/30); Task 3 from 73% (n=22/30); Task 4 from 77% (n=23/30); and Task 5 and Task 6 from 90% (n=27/30) of the participants.
For Task 6 (
The content validity study questions can be found in
Sources of validity evidence for the minimally invasive surgery training–virtual reality.
Source of validity evidence | Studies |
Content evidence | [ |
Internal structure | [ |
Relationship to other variables | [ |
Consequences | [ |
Simulation as a tool for learning psychomotor skills in MIS has become a new model for education in surgery. The use of human or animal cadavers is becoming increasingly controversial for learning surgical maneuvers [
Simulators for psychomotor skills learning in MIS are classified into mechanical, hybrid/augmented reality, or VR [
The development of SIMISGEST-VR was based on DBR principles. It was a
To develop this study’s 3D virtual environment, the researchers adopted the principle of low fidelity, given that the model is envisaged for basic psychomotor skills learning. The term fidelity refers to the extent to which a simulation imitates reality (in the case of surgical simulation, the anatomy) and is considered a critical variable in the design of simulators. However, this statement is not necessarily completely true, as for novice learners, low-fidelity models that reproduce the essential constructs of a procedure allow a faster and more cost-effective learning curve to be achieved [
The tasks were adapted from the MIST-VR, which is the only laparoscopic VR trainer that can act as a standard because it is the sole surgical VR system that has been reasonably validated [
Performance evaluation is a fundamental part of the learning process and is essential for certification. To obtain an objective evaluation of performance, the simulator should define metrics that must be valid, accurate, and relevant in terms of the procedure that is being taught. Evaluation using metrics and effective feedback are the most important elements of effective learning in a simulation environment [
Feedback is essential [
The design of the hardware components aimed to simulate the movements made by the surgeon during MIS. These movements are defined by the physical characteristics of the devices and, therefore, require the design of mechanical support devices that simulate the fulcrum effect (entry portals), add friction to the movements of the forceps, and limit arm movement during the performance of the tasks without interfering with the reading of the instrument movements by the LMC [
The VR or augmented VR simulators currently available in the market are not portable, and their cost ranges from US $2000 to US $100,000 (with annual maintenance costs of US $25,000) for a haptic VR simulator. The LMC costs approximately US $130, plus a further US $70 for the hardware elements, adding up to a total cost of approximately US $200 for the SIMISGEST-VR, software costs excluded.
The second aim of this study was to evaluate fidelity to the criterion and a content validity study. Validity refers to the quality of the inferences, claims, or decisions taken from the scores given by an instrument, not the instrument itself. Validation for its part is a process through which the evidence that supports the quality, significance, and utility of the decisions and inferences that can be made from the scores provided by the instrument is drawn together and evaluated [
Although it has been deemed that
Fidelity to the criterion evaluates to what point the test reflects the real-life situation, whether the simulator represents what it is supposed to represent (the realism of the simulator) or the extent to which a questionnaire or other measurement reflects the variable to be measured [
In this study, the evaluation of fidelity to the criterion provided feedback on the initial design, and this was how the 3D virtual environment was redesigned until a
In all the items evaluated for fidelity to the criterion, most of those surveyed assigned scores of 4 or 5. There were no significant differences between the expert and referent groups (level of training) when rating fidelity to the criterion. The lowest scores were obtained for the item about the relevance (n=9/30, 3% of participants), the representation of the movements of the physical forceps in the virtual environment (n=9/30, 3%), and for the fulcrum effect (n=3/30, 10%).
The latest standards on validity and validation refer to sources of validity evidence, rather than distinct types of validity. Validity therefore refers to the degree to which the evidence and theory support the interpretations of test scores for the proposed uses of tests [
Evidence based on test content is an issue of representation and may be obtained from an analysis of the relationship between test content and the construct that is intended to be measured. In this study, the test content refers to the simulator’s 6 specific tasks. Evidence can be obtained from logical or empirical analyses of how test content represents domain content and of the relevance of domain content to the proposed interpretation of test scores. Evidence may also come from experts’ opinions on the relationship between the different test items and the construct when assessing whether the test contains the meaningful steps, skills, and materials used in the actual procedure [
The question is, does the simulator realistically teach what it should teach? In other words, does the instrument represent all the ways in which it can be used to measure the content of a given construct? [
This type of validation is highly recommended in the practice of DBR during the design phase of the
The tasks within the surgical simulation should fulfill 3 criteria: objectivity, clarity, and completeness. To be objective, the definition of the task should refer to observable characteristics of the behavior; for it to be clear, the task should be unambiguous so that it can be read, understood, and reproduced equally by different observers; and finally, to meet the criterion of completeness, the definition of the task should delineate its start and end and make it clear when it was completed [
In this study, the 6 skill tasks were chosen for two main reasons: (1) these tasks are well-validated in many clinical studies [
The vast majority of study participants considered that the SIMISGEST-VR was a useful tool for the development of hand-eye coordination and depth perception, with ratings of 4 and 5 on the Likert scale. Similarly, there was consensus about the capacity of the simulator to teach basic psychomotor skills and to reflect the basic steps in MIS. All the respondents considered the metrics to be adequate and envisaged that the simulator could become a solution to achieve ubiquitous learning of basic psychomotor skills in MIS.
In terms of the specific rating for each of the 6 tasks, this varied between 3.97 and 4.53. The participants considered all the items of the SIMISGEST-VR training system as good to excellent.
Finally, the study of fidelity to the criterion and content validity must be proven in the design stage of the artifact, before the criterion (concurrent and predictive) and construct validity (convergent and discriminative) can be confirmed. The evaluation of fidelity to the criterion, although somewhat subjective, is a necessary assessment during the initial phase of any high-stakes test construction and in this study, within the context of DBR, in the design phase of prototypes that will give a
The
Regarding the representation of the construct, in this study there was an underrepresentation—when compared with the learning models based on training boxes—referring to the
There are, however, limitations to this study. The sample size of this study was one of availability and, for the simulator to be portable and allow ubiquitous learning, the researchers disregarded some ergonomic principles applied to MIS [
The researchers of this study are currently conducting another study to show validity evidence for the
Once the metrics and the results of the performance scores have been validated as a useful tool for learning basic psychomotor skills in MIS, a model will be obtained to enable ubiquitous learning in MIS and preoperative warm-up by using the 3D reconstruction of patient images [
The large size and elevated costs of VR simulators currently available in the market prohibit their use in the operating theater. A portable, low-cost simulation solution, such as the SIMISGEST-VR, would allow surgeons to perform preoperative warm-up exercises anytime, anywhere (ubiquitous learning). In addition, the researchers aim to enable a surgeon to perform warm-up exercises based on 3D reconstructions of preoperative images of a specific patient, thus, practicing the procedure before performing the actual surgery. This could take place the night before in the surgeon's home or the operating theater on the day of the surgery [
This study demonstrated the feasibility of a portable, low-cost, gesture-based, functional simulator (SIMISGEST-VR) for learning basic psychomotor skills in MIS.
The results of the evaluation of fidelity to the criterion and content validity showed overall positive scores, which indicates that the SIMISGEST-VR would be acceptable to both the expert group and referent group as a training and learning device (including at home) to achieve ubiquitous learning in MIS.
The participants in the study agreed that content validity was acceptable, accurate, and representative in the field of basic psychomotor skills learning in MIS.
Application forms of the demographic survey, fidelity to the criterion, and content validity surveys.
Results of the fidelity to the criterion survey.
Results of the content validity survey.
minimally invasive surgery
minimally invasive surgery training—virtual reality
Leap Motion Controller
design-based research
gesture-mediated simulator for minimally invasive surgery—virtual reality
All the authors contributed substantially to the study conception and design, data analysis, and interpretation of the findings and manuscript drafting. Fernando Álvarez López participated in the collection and assembly of data. Francesc Saigí-Rubió is the guarantor of the paper. All the authors have read, revised, and approved the final manuscript.
None declared.