A prospective, simulation-based trial was conducted in the department of simulation in health care iLumens (a multidisciplinary university medical laboratory focused on digital health education) in Paris Descartes University (France). The study was approved by the ethics committee of our institution (CENEM 2019-15-DD) and registered in ClinicalTrials.gov [NCT03884114].

Participants were fifth-year medical students from the French faculty of medicine Paris Descartes who participated voluntarily. They were part of two classes and went to the department of simulation in March and June 2019 after their 3-month pediatrics courses which included a 2-hour course on pediatric asthma exacerbation management and within 15 days after their pediatrics exams. Recruitment was done by emails and during their pediatric exams. Written consent was obtained, and demographic data were collected from participants.

The study design is presented in Figure 1. Participants were successively evaluated on the same scenario of a moderate asthma exacerbation occurring in a private medical practice using three different modalities (HFS, SG, and MCQ) for a maximum duration of 12 minutes for each modality. Participants started by watching a 5-minutes tutorial video on the features of the high-fidelity manikin SimBaby (Laerdal Medical AS) used and its environment. They were then individually evaluated on the management of a moderate asthma exacerbation scenario on HFS. No debriefing was provided, and participants were invited to watch a 3-minute video on the features of the SG (Effic’Asthme, iLumens, and Dowino). The second evaluation was conducted on the same scenario using the SG. Again, no debriefing was provided, and the third evaluation was conducted on the same scenario using an MCQ. Finally, participants completed a questionnaire on their characteristics, followed by a second questionnaire about their satisfaction with each evaluation modality and ending with an oral debriefing.

A single scenario was used for all three modalities. It corresponded to a moderate exacerbation of asthma in a child aged 12-months who did not respond to short-acting beta-agonists (SABAs). A full description of the scenario is presented in Multimedia Appendix 1 (Table S1). Participants were expected to administer the emergency treatment (salbutamol, SABAs) to the child with the correct inhalation technique and dose, repeat this administration after no improvement was noted, repeat this administration again adding oral corticosteroids, and finally refer the child to the nearest pediatric emergency department.

The first evaluation modality studied was HFS. As it was impossible to assess the students on their ability to manage an asthma exacerbation with a real patient, we considered HFS the gold standard assessment method that would best reflect students’ clinical competence. The Simbaby manikin used can reproduce many signs of an asthma exacerbation. In the simulation room, participants could use the same items as in the SG (Figure 2): an emergency treatment (salbutamol, SABAs), a controller treatment (fluticasone), an asthma spacer with a facial mask, oral steroids in tablets (prednisone), a glass of water, paracetamol, saline nose drops for nasal airway clearance, and a phone. All actions performed by the participants were video recorded.

The second modality used was the SG Effic’Asthme, a mobile app displayed on a tablet. This SG was developed by iLumens at Paris Descartes University and Dowino to train parents to recognize and manage different scenarios of asthma exacerbation in their children [16]. When connecting to Effic’Asthme for the first time, the user is asked to enter information to create a child avatar. The home screen then proposes the following sections: asthma action plan, asthma attack log, training, documents, and doctor (Figure 3). The training section corresponds to the SG. After completing a tutorial, the player can choose among 6 scenarios of asthma exacerbations with varying levels of severity, all taking place in the virtual bedroom of the child. A video presenting the main features of Effic’Asthme is available (Multimedia Appendix 2). Each scenario starts with a short briefing of the situation. A message then invites the user to carefully observe the child’s avatar with the possibility of zooming and rotating the child in 3D to detect any sign of respiratory distress and to listen for a cough or wheezing. Depending on the child’s condition, the user needs to choose the actions to be performed to manage the asthma exacerbation appropriately among different panels (Figure 4 left). One of the main outcomes is to check the administration technique of the emergency treatment (salbutamol/albuterol) to the child (Figure 4 right). The user can use his/her fingers to remove the cap of the spray, shake it, insert it into the spacer, place the spacer on the child’s face, press the spray to administer one puff, and wait for the number of breaths they deem necessary between each puff. Once the scenario is completed, an automated, point-by-point debriefing is provided. Points are awarded for actions performed correctly, and an overall success rate of the mission out of 100% is given.

For the study, Effic’Asthme was diverted from its initial use to assess students’ clinical competence. This SG was chosen because it allowed the evaluation of several dimensions of clinical competence such as knowledge and clinical reasoning and technical skills such as the administration technique of the inhaled emergency treatment.

The third assessment modality was an MCQ designed for the study (Table S2 in Multimedia Appendix 1). As with the HFS and SG, the MCQ started with a short briefing followed by 15 questions on the management of the same scenario of moderate asthma exacerbation.

For each evaluation modality, the investigators completed a standardized checklist (common for the three modalities, Table S3 in Multimedia Appendix 1). This checklist had been previously validated [16,17]. The checklist included 19 items. Each item was rated 0 or 1, depending on whether or not the student had performed the correct action when evaluated on the HFS or SG or provided the correct answer on the MCQ, with a maximum score of 19.

The main outcome was the degree of correlation between the actions performed for each item between the SG and HFS on the one hand and the concordance of the answers on the MCQ and the actions performed on the HFS on the other hand. For each item rated 0 or 1 for the modalities SG and HFS or MCQ and HFS, the Cohen kappa coefficient was calculated to estimate the concordance of the actions performed and answers provided at the level of the population studied. The median kappa coefficients on the 19 items were calculated and compared between the SG/HFS group and the MCQ/HFS group.

Secondary outcomes included the comparison between the median checklist scores for the HFS, SG, and MCQ modalities; comparison of the dispersion of the checklist scores for each modality; and participant satisfaction with the three assessment modalities.

The sample size calculation was based on the following assumptions: 2-tailed alpha of .05, power of 80%, a median kappa coefficient estimated at 0.4 for the MCQ/HFS comparisons, and an expected median kappa coefficient of 0.7 for the SG/HFS comparisons. The calculation suggested that at least 39 participants should be enrolled [18].

Data were collected in Excel (Microsoft Corp) and analyzed with Prism software version 5.03 (GraphPad). Median values and interquartile ranges were reported. Cohen kappa coefficients for each item of the checklist between the HFS/SG and HFS/MCQ groups were calculated. The median kappa coefficients for each group were compared using a Mann-Whitney U test to compare the correlation of actions and answers between these two groups. The Spearman rank correlation coefficient was used to compare the median checklist scores between the three modalities. Finally, the dispersion of scores between the SG and MCQ modalities was studied by comparing variances using a modified Levene test, and participant satisfaction for each modality was compared using a student t test.

HFS and OSCE, the best available modalities for assessing the clinical competence of medical students without any risk to the patient, are almost impossible to implement on a large scale due to their high consumption of time and human resources. Thus, many faculties use MCQ as a default solution for large-scale examinations, with the risk of favoring students who are highly competent to respond to MCQs but unable to manage real-life emergency situations. This study demonstrates that Effic’Asthme, an SG created to teach about the management of pediatric asthma exacerbations, better reflects the actions of fifth-year medical students on HFS than an MCQ on the same clinical case. We propose in Table 3 a summary of the domains of clinical competence that can be assessed through an SG such as Effic’Asthme, in comparison with MCQ and HFS. As illustrated, if further studies confirm these findings, SG may become an intermediate solution between HFS and MCQ, evaluating students’ clinical competence further than MCQ while remaining widely deployable, unlike HFS.

Contextualizing our results within the medical literature is limited by the paucity of data on the comparison of SG with other assessment modalities. To our knowledge, only the study by Adjedj et al [15] compared the SG and MCQ modalities for the evaluation of medical students, on a clinical case of cardiology. They found greater variability in the scores obtained on the SG than in the scores obtained on the MCQ and concluded that SG had the potential to better rank students [15]. However, because their study did not use a gold standard as ours did, they were unable to determine which of the two assessment modalities was more relevant for evaluating students. Therefore, our study goes further in showing that SG is indeed a more relevant assessment modality than MCQ for the evaluation of the management of a child’s asthma exacerbation, as it better reflects the actions performed on HFS. Surprisingly, whereas the analysis, item by item, of the actions performed/answers provided revealed a higher level of concordance between the SG/HFS modalities than between the MCQ/HFS modalities, this difference was no longer statistically significant when the total checklist scores were compared (P=.11). The most likely explanation is a lack of power because our sample size was not calculated on this outcome.

After considering the SG Effic’Asthme as a new evaluation modality, its acceptability and cost should be scrutinized. The results of the survey conducted with the medical students who participated in this study indicate that students support the use of this evaluation modality for their exams, as reported in other studies [20,21]. It would be interesting to collect the opinions of the members of the faculty to see if they are consistent with those of students. Clearly, faculty members will need to estimate the costs of this new assessment modality. Developing SG is costly: their development is costly. For example, the development of Effic’Asthme cost €135,000 (US $160,000) [22]. This cost could be reduced in the future by using an interuniversity platform integrating all the necessary elements to create a multitude of scenarios in the same way that high-fidelity simulators are provided with a software allowing to design hundreds of different situations.

A limitation of the SG revealed by this study was the need to integrate a tutorial. Indeed, in our study, many participants did not perform actions such as shaking the inhaler or removing its cap, although they had performed these actions on the HFS and the MCQ. This suggests that they thought it was impossible to perform these actions on the SG. Thus, based on the results of this study, a key recommendation for future developers of SGs is to develop a neutral tutorial before the assessment part of their SG.

The study has several limitations. First, it can be argued that the order of assessments (HFS then SG then MCQ) may have favored higher scores on the MCQ modality. Initially, we considered a crossover trial: after their evaluation with the HFS modality, students would have been randomized into two groups, one group starting by the SG session and ending with the MCQ modality as in this study, and the other starting by the MCQ modality and ending with the SG session. However, the MCQ required students to be provided several cues (eg, when asking for the dose of SABAs or oral corticosteroids, it was implied that SABAs and oral corticosteroids should be administered). This would have strongly biased the next evaluation on SG. By contrast, the HFS and SG modalities did not influence the student on specific actions to be taken because no indication was provided during or after the sessions, and we judged that there was no reason for students to perform better over the course of the different assessments without being provided any indication between them. Indeed, it was demonstrated that no gain in knowledge can be achieved when no debriefing is provided following a simulation session [23]. The use of HFS as a gold standard for assessing medical students’ competence is also debatable. HFS is probably not a modality that is perfectly representative of the care that would have been provided by the students under real-world conditions, but it corresponded to the only option that we have found to be both feasible and ethical. Finally, the fact that the study was monocentric and evaluated only volunteer students on a single scenario limits the generalizability of the results. Despite these limitations, this trial is the first to compare SG and MCQ as assessment methods for medical students.

In conclusion, the SG used in this study better reflected the clinical competence of students on HFS than an MCQ on the same clinical case of pediatric asthma exacerbation. Its use as an assessment method was appreciated by the students. If further studies confirm these results, SG could become an interesting compromise for the evaluation of medical students between the cheap but limited assessment allowed by MCQs and the comprehensive but highly expensive assessment allowed by HFS.