This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The objective structured clinical examination (OSCE) is a test used throughout Spain to evaluate the clinical competencies, decision making, problem solving, and other skills of sixth-year medical students.
The main goal of this study is to explore the possible applications and utility of portable eye-tracking systems in the setting of the OSCE, particularly questions associated with attention and engagement.
We used a portable Tobii Glasses 2 eye tracker, which allows real-time monitoring of where the students were looking and records the voice and ambient sounds. We then performed a qualitative and a quantitative analysis of the fields of vision and gaze points attracting attention as well as the visual itinerary.
Eye-tracking technology was used in the OSCE with no major issues. This portable system was of the greatest value in the patient simulators and mannequin stations, where interaction with the simulated patient or areas of interest in the mannequin can be quantified. This technology proved useful to better identify the areas of interest in the medical images provided.
Portable eye trackers offer the opportunity to improve the objective evaluation of candidates and the self-evaluation of the stations used as well as medical simulations by examiners. We suggest that this technology has enough resolution to identify where a student is looking at and could be useful for developing new approaches for evaluating specific aspects of clinical competencies.
The objective structured clinical examination (OSCE) is an evaluation of medical students that aims to assess candidates’ skills and attitudes in certain clinical situations. This goal is different from that of typical written exams, which primarily evaluate knowledge. Basically, evaluating clinical competencies entails objective measurement of whether a candidate has correctly used and applied theoretical knowledge [
The OSCE incorporates diverse evaluation instruments, which are arranged in successive stations that simulate real clinical situations [
In Spain, various medical schools have not implemented this testing method uniformly, with universities in Catalonia—as with many medical schools worldwide [
Recently, the National Deans Conference in Spain approved a 20-station OSCE that all sixth-year medical students must take. The general characteristics of the test are practical character; oriented toward evaluating the professional competencies of the candidate relative to the specific competencies of the medical degree, established by the Order ECI/332/2008 (published in the Official State Gazette on February 15, 2008); performed through the resolution of clinical cases; and having the objective of demonstrating clinical skills.
The OSCE explores diverse areas of evaluation through a range of methodologies. The stations include simulated and standardized patients, mannequins, short-answer questions, performance of complementary examinations adjusted to the particular case, clinical report writing, structured oral examinations (SOEs), skills and procedures, computer functions, and simulators. The OSCE entails a considerable collective effort both for candidates and examiners; however, several studies have also reported good cost-effectiveness [
Video recording for clinical training purposes is already used successfully, including to evaluate procedures, techniques, or diagnostic tests [
With this in mind, we designed a feasibility study to assess the real possibilities of eye tracking as a tool for teaching and evaluation in the OSCE, according to each station model. To the best of our knowledge, this is the first study to use eye-tracking glasses in each of the 20 OSCE stations. Our main goal is to identify whether this technology can be used in an examination as extensive as the OSCE. In addition, we analyzed the opportunities for evaluating students according to the type of station, the usefulness perceived by the teachers, and the teaching opportunities for students. The hypothesis was that this technology is useful in teaching, specifically in the OSCE.
This is a descriptive study on the use of eye-tracking glasses during the OSCE. We carried out an OSCE of 117 sixth-year medical students at the Medical School of Miguel Hernández University Spain in June 2017. The OSCE consisted of a circuit of 20 stations or situations, and the candidates had to move consecutively through all of them, spending 9 minutes on each, with 2-minute rest periods between each new station. The stations and the skill areas included were history taking, physical examination, doctor-patient communication, clinical report writing, clinical judgement, technical skills, preventive activities, and ethical-legal issues (
The exam was carried out in parallel rounds of 23 candidates each. All rounds were conducted on the same day. Two rounds were held in the morning and three rounds in the afternoon (one with 25 students). Forty consultation areas were prepared in line with the needs of the specific stations, each equipped with a computer program created for candidate evaluation. The simulated patient or examiner completed a checklist of evaluation items for the skill area being assessed. Exams were continuous, with rotations communicated through a computer program with speakers. The exams lasted 4.5 hours. Candidates were not permitted to have any electronic items with them (including mobile phones or watches), and all had a white coat, stethoscope, pencil or pen, and two blank sheets of paper.
The portable eye trackers used were the Tobii Glasses 2 [
The videos for each station were edited to eliminate the waiting periods and to include only the time period from which the participants read the case being evaluated prior to entering the consultation area. The audio recordings at each station were also homogenized so that the volume levels were approximately the same for all stations.
Station, type of station, and skills map for the objective structured clinical examination.
Station number | Station | Type | HTa | CEb | TSPc | CSd | CJe | DPf | IRg | ELh | Total |
1 | MSPi of the digestive tract | SPj | 65.5 | 24.1 | 0.0 | 10.3 | 0.0 | 0.0 | 0.0 | 0.0 | 100 |
2 | MSP of the digestive tract | Report | 0.0 | 0.0 | 0.0 | 0.0 | 34.8 | 65.3 | 0.0 | 0.0 | 100 |
3 | Infectious disease pathology | SOEk | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 90.0 | 10.0 | 100 |
4 | Emergency medicine | M-Pl | 0.0 | 35.0 | 55.0 | 0.0 | 0.0 | 0.0 | 0.0 | 10.0 | 100 |
5 | Legal medicine | Report | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 38.1 | 61.9 | 0.0 | 100 |
6 | Emergency | SP | 0.0 | 96.8 | 0.0 | 3.2 | 0.0 | 0.0 | 0.0 | 0.0 | 100 |
7 | MSP of the nephrourinary system | M-P | 0.0 | 0.0 | 55.0 | 0.0 | 35.0 | 0.0 | 0.0 | 10.0 | 100 |
8 | Psychiatry | SP | 22.6 | 35.5 | 0.0 | 12.9 | 29.0 | 0.0 | 0.0 | 0.0 | 100 |
9 | MSP of the musculoskeletal system | M-P | 0.0 | 57.2 | 28.6 | 0.0 | 14.3 | 0.0 | 0.0 | 0.0 | 100 |
10 | Pediatrics | SOE | 0.0 | 0.0 | 0.0 | 0.0 | 42.1 | 0.0 | 47.4 | 10.5 | 100 |
11 | Infectious disease pathology | SP | 43.3 | 13.3 | 0.0 | 10 | 33.4 | 0.0 | 0.0 | 0.0 | 100 |
12 | MSP of the endocrine system | SOE | 20 | 0 | 30 | 0 | 10 | 0 | 40 | 0 | 100 |
13 | Surgery | M-P | 0.0 | 0.0 | 83.3 | 0.0 | 0.0 | 0.0 | 12.5 | 4.2 | 100 |
14 | Gynecology and obstetrics | SP | 50.0 | 0.0 | 0.0 | 15.0 | 25 | 0.0 | 10 | 0.0 | 100 |
15 | Gynecology and obstetrics | M-P | 0.0 | 15.8 | 26.3 | 0.0 | 47.4 | 0.0 | 10.5 | 0.0 | 100 |
16 | Pediatrics | SP | 38.5 | 30.8 | 0.0 | 11.5 | 11.5 | 0.0 | 7.7 | 0.0 | 100 |
17 | MSP of the respiratory system | Report | 0 | 0 | 0 | 0 | 70 | 30 | 0 | 0 | 100 |
18 | MSP of the cardiovascular system | SP | 44 | 36 | 0 | 12 | 8 | 0 | 0 | 0 | 100 |
19 | MSP of the cardiovascular system | Report | 0 | 0 | 0 | 0 | 100 | 0 | 0 | 0 | 100 |
20 | Microbiology and legal medicine | Report | 0.0 | 0.0 | 0.0 | 0.0 | 38.9 | 0.0 | 22.2 | 38.9 | 100 |
aHT: history taking.
bCE: clinical examination.
cTSP: technical skills and procedures.
dCS: communication skills.
eCJ: clinical judgement, diagnostic and therapeutic management plan.
fDP: disease prevention and health promotion.
gIR: interprofessional relationships.
hEL: ethical issues—legality and professionalism.
iMSP: medical and surgical pathology.
jSP: standardized patients.
kSOE: structured oral examination.
lM-P: mannequin-procedure.
The analysis was performed using Tobii Glasses Analysis Software and included a quantitative study of gaze points and the order that they appeared for each subject. We also used heat maps for visualization of eye tracking. Thus, we identified the areas of interest in each station and the percentage of the total area that they occupied within the image as a whole. Using these percentages, we calculated the participants’ most frequent gaze locations, the number of gazes, and the visual itinerary (ie, the order that the gazes occurred). We also measured pupil dilation, but we found that it was extremely difficult to control luminance and maintain stable lighting conditions in our 20 different experimental conditions. Hence, we discovered that this particular measure was not appropriate for this type of study. Results were exported to Excel (Microsoft Corp) for statistical analysis.
Portable eye tracker used.
Finally, each of the station examiners (N=30) viewed the videos and completed a questionnaire to assess the usefulness of the eye tracker for each type of station in the visualization and interpretation of complementary tests, the assessment of the attitude of simulated patients and examiners, the evaluation of the candidate, the characterization of empathy and eye contact, the assessment of the design of the physical space where the exam was performed, the stimulation of ideas on training possibilities for examiners, and the external evaluation of the exam. Each of these questions was scored according to the usefulness of the device as follows: 0 not at all useful, 1 a little useful, 2 somewhat useful, and 3 very useful.
The Research Ethics Committee of Miguel Hernandez University approved this study (DMC.JRR.01.17). Verbal consent was obtained from the study participants.
The OSCE consisted of 7 stations with simulated patients, 5 testing technical skills and procedures with or without mannequins, 5 testing candidates’ abilities to draft a clinical report, and 3 SOEs on practical clinical situations. Of the 20 stations, we obtained useful recordings of 16 (80%). No recordings were made for 3 of the stations because the specific tasks involved writing reports (stations 5, 19, and 20). Furthermore, we encountered technical problems in the recordings associated with station 7 (possibly due to the battery); therefore, we discarded these data.
We evaluated 7 stations with simulated patients (stations 1, 6, 8, 11, 14, 16, and 18). The mean time for history taking was 242 (SD 28.4) seconds. In this group of stations, the eye‑tracking device provided a wealth of usable data, demonstrating the extent to which the candidate
In station 16, this pattern was of special interest, as it involved a simulated mother with a pediatric mannequin. The areas of visual interest were the faces of the mother and the infant (
The areas of visual interest were the faces of the mother and the infant.
We evaluated 5 stations (1, 4, 7, 9, and 15) associated with the assessment of technical skills. Given that the students were constantly moving in these stations, for example, in the station associated with cardiopulmonary resuscitation (CPR) or in the station to evaluate suturing skills, it was difficult to analyze the results. Analysis of these data, therefore, required manual coding of the video to quantify the steadiness of the gaze [
In drafting clinical reports, for example, on electrocardiograms, x-rays, or other images, the eye tracker was not useful, as it only showed the candidate writing. However, in the stations evaluating a diagnostic test (for example, a chest x-ray), we were able to measure the time the candidates spent evaluating the image and the potential relationships between the focus of attention and the student’s interpretation of the clinical case.
The time spent and location of the gaze by the medical student looking at chest x-ray.
The time spent and location of gaze by the medical student looking at an image of skin lesions on a lower limb.
Usefulness of portable eye-tracking devices in the OSCE by type of station for 30 station examiners.
Opportunities for eye tracking | Type of station, modea | |||
|
Standardized patients | Mannequin/procedure | Report | SOEb |
Evaluation of complementary tests | 0 | 1 | 2 | 1 |
Evaluation of simulated patient/examiner | 3 | 3 | 0 | 2 |
Re-evaluation of candidate | 2 | 3 | 0 | 1 |
Empathy/eye contact | 3 | 1 | 0 | 1 |
Design of OSCEc consultation area | 3 | 3 | 0 | 0 |
Candidate preparation for OSCE | 2 | 3 | 1 | 2 |
External evaluation of OSCE | 3 | 3 | 0 | 2 |
aLikert scale: 0 not at all useful, 1 a little useful, 2 somewhat useful, 3 very useful.
bSOE: structured oral examination.
cOSCE: objective structured clinical examination.
In contrast, the eye tracker was
Our study shows that portable eye-tracking is an applicable and useful tool in the OSCE. In the standardized patient stations, mannequins or pictograms of the procedure were useful in many aspects. Herein is where teachers see more possibilities for their use. In contrast, in the stations where the students had to write reports, the eye tracking was not particularly useful. In addition, since this was the first time this technology was implemented in a complete OSCE, we found that preparation of batteries sufficient for the entire test is essential (recording of one station was lost due to battery issues).
OSCE testing has now been implemented in all the medical schools in Spain, with some programs dating back more than 20 years [
The use of portable eye-tracking technology has been introduced in medicine more recently [
Our preliminary results show a wide range of possibilities for future research. The technology may help evaluators to objectively measure the empathy shown by the candidate in the stations with simulated patients by characterizing the features of the candidate’s gaze (eg, establishment of eye contact or steadiness). Although the OSCE is intended as an objective test, some elements are assessed with a certain degree of subjectivity, such as the scores associated with the candidate’s treatment of patients, nonverbal communication, and conversation and empathy [
Until now, in the stations assessing candidates’ interpretation of image prompts (electrocardiogram, x-ray, photos, etc), we could not know whether candidates responded correctly to the questions as a result of an adequate systematic approach that included a revision of all relevant points or whether their responses were due to chance or previous knowledge of the answer [
The usefulness of eye tracking varied according to the station, whether these dealt with image-based clinical reports, images shown to the candidate, simulated patients (as a method to quantify doctor-patient empathy), or procedures or mannequins in the re-evaluation of the candidate. Thus, in the stations using images, we believe that it may be more practical to use optical tracking by means of a device placed directly on the monitor showing the image. As in other studies [
On the other hand, the OSCE is stressful for medical students. To help them to prepare this evaluation, universities have created a study guide [
We are fully aware that this is a proof-of-concept study and has important limitations. Furthermore, we were not able to compare the results of several students, as eye tracking is an expensive technology, and we only had a few devices available. Nevertheless, our preliminary results suggest that portable eye-tracking devices offer a number of opportunities in the field of OSCE evaluation, including the design, set up, and self-evaluation of the examiners at each station. Moreover, these devices may also provide insight into methods to improve the evaluation of candidates during the exams, particularly in the stations assessing subjective elements. Nevertheless, further studies are still needed.
Sample video captured during the investigation.
cardiopulmonary resuscitation
objective structured clinical examination
structured oral examination
The authors are grateful for the collaboration of the people who participated in the study. The authors disclosed no receipt of any financial support for the research. The data sets used during or analyzed during this study are available from the corresponding author on reasonable request.
MDGM contributed to the design, data acquisition, data analysis, and interpretation; critically revised the manuscript; gave final approval; and agrees to be held accountable for all aspects of the work, ensuring integrity and accuracy. FSF contributed to the design, data acquisition, data analysis, and interpretation; critically revised the manuscript; gave final approval; and agrees to be held accountable for all aspects of the work, ensuring integrity and accuracy. EF contributed to the design, data acquisition, data analysis, and interpretation; critically revised the manuscript; gave final approval; and agrees to be held accountable for all aspects of the work, ensuring integrity and accuracy. JMRR contributed to the conception, design, and interpretation; drafted and critically revised the manuscript; gave final approval; and agrees to be held accountable for all aspects of the work, ensuring integrity and accuracy.
None declared.