This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
To avoid misuse of personal protective equipment (PPE), ensure health care workers’ safety, and avoid shortages, effective communication of up-to-date infection control guidelines is essential. As prehospital teams are particularly at risk of contamination given their challenging work environment, a specific gamified electronic learning (e-learning) module targeting this audience might provide significant advantages as it requires neither the presence of learners nor the repetitive use of equipment for demonstration.
The aim of this study was to evaluate whether a gamified e-learning module could improve the rate of adequate PPE choice by prehospital personnel in the context of the coronavirus disease (COVID-19) pandemic.
This was an individual-level randomized, controlled, quadruple-blind (investigators, participants, outcome assessors, and data analysts) closed web-based trial. All emergency prehospital personnel working in Geneva, Switzerland, were eligible for inclusion, and were invited to participate by email in April 2020. Participants were informed that the study aim was to assess their knowledge regarding PPE, and that they would be presented with both the guidelines and the e-learning module, though they were unaware that there were two different study paths. All participants first answered a preintervention quiz designed to establish their profile and baseline knowledge. The control group then accessed the guidelines before answering a second set of questions, and were then granted access to the e-learning module. The e-learning group was shown the e-learning module right after the guidelines and before answering the second set of questions.
Of the 291 randomized participants, 176 (60.5%) completed the trial. There was no significant difference in baseline knowledge between groups. Though the baseline proportion of adequate PPE choice was high (75%, IQR 50%-75%), participants’ description of the donning sequence was in most cases incorrect. After either intervention, adequate choice of PPE increased significantly in both groups (
Among prehospital personnel with an already relatively high knowledge of and experience with PPE use, both web-based study paths increased the rate of adequate choice of PPE. There was no major added value of the gamified e-learning module apart from preserving participants' confidence in their ability to correctly use PPE.
Adequate use of personal protective equipment (PPE) is of paramount importance to ensure health care workers’ safety and to avoid shortages of such equipment in the context of the coronavirus disease (COVID-19) pandemic [
To avoid misuse of PPE, effective communication of the corresponding guidelines to frontline health care workers is necessary [
The purpose of this study was to evaluate whether a specifically designed gamified e-learning module [
This was an individual-level, stratified, randomized, controlled, quadruple-blind (investigators, participants, outcome assessors, and data analysts) closed web-based trial (
Study design.
The regional ethics committee issued a “Declaration of no objection” (Req-2020-00374) as the population studied was not considered vulnerable according to the Swiss federal law on human research [
The study took place in April 2020 in Geneva, Switzerland; the detailed organization of local prehospital emergency services has already been described elsewhere [
Apart from the physician-staffed prehospital medical service, there are seven different ambulance companies in Geneva, two of which belong to public organizations while the remaining five are privately owned. As each company has its own continuous education structure as well as its own equipment, medical devices, and protocols, one cannot expect all paramedics to share the same knowledge level regarding all aspects of prehospital emergencies.
An online platform [
A previously described gamified e-learning module created under Storyline 3 (Articulate Global) was used in this study [
Two quizzes were created by BG and L Suppan: a preintervention quiz designed to establish the participants’ baseline knowledge regarding PPE, their use and indication, and a postintervention quiz to assess whether these parameters had changed. Both quizzes contained 10 closed questions, either multiple choice or multiple answer. Questions designed to assess PPE choice were preceded by short clinical scenarios. Each quiz was displayed over 5 pages. The number of questions was limited to reduce attrition. These quizzes were tested and validated by all coinvestigators.
Consistency of specific “free-text” questions, such as age, was checked by means of regular expression (“regex”) rules. All answers were automatically checked for completeness by the system before participants were allowed to proceed to the next page. Custom validation messages were displayed to inform users who had not answered a question. Participants were not allowed to correct or review their answers once a page had been completed.
Chief ambulance officers of all services were asked to provide one of the investigators (L Suppan) with a list of all the professional email addresses of their EMTs, paramedics, and emergency physicians. All the email addresses received from these officers were included.
Email addresses of ambulance drivers were excluded as these drivers usually only deal with interhospital transfers and almost never don PPE. Senior specialist emergency physicians were also excluded, as they are few in number and are usually involved in the writing of the guidelines or in the creation of the learning material; in addition, some of them are authors of this study. Finally, the email addresses of the paramedics who participated in the creation of this study or the learning material were also excluded.
Before performing a 1:1 randomization, stratification was achieved according to professional status (EMTs, paramedics, and emergency physicians). Email addresses were then sorted according to alphabetical order, and an investigator (MS) who did not have access to the email addresses database was given the number of participants by category and performed the randomization using a computer-generated table. The randomization key was then combined with the list of email addresses and entered in the mailing component by the only investigator who had access to the system (L Suppan). As the list of email addresses and allocations were solely present in an encrypted database, all other investigators were blinded as to group allocation.
Individual emails that were identical for all participants except for the unique links that pointed to one of the two study paths were sent on April 13, 2020 (
Apart from the survey link, the emails contained information regarding the study length and objectives as well as a short data protection statement. Participants were informed that they would be presented with the most recent version of the prehospital COVID-19 guidelines as well as with an e-learning module, though the order in which these materials would be shown was not explained. They were informed that, by clicking on the survey link, they consented to participate in the study and were provided with the names and electronic addresses of five investigators (BG, EG, L Stuby, L Suppan, and PC), whom they could contact at any time. As collected data were irreversibly anonymized, it was impossible for users to ask for their own answers to be deleted once the survey had been completed.
To improve the response rate, the chief ambulance and medical officers of all companies were asked to encourage their paramedics, EMTs, and emergency physicians to participate in the study. Participation reminders were also sent to all prehospital personnel along with a daily COVID-19 information newsletter.
Participation was not mandatory. No monetary incentive or prize was offered to the survey participants. As the e-learning module was akin to a continuous education session, participants were informed before beginning the survey that they would be able to print a continuous education certificate upon completion. As the certificate component was independent from both the mailing and the survey components, participants were ensured they could generate the certificate without their identity being disclosed.
After clicking on the survey link, a welcome screen containing detailed information about the study was displayed. This welcome screen was identical for both groups and, similarly to the email messages, did not convey any information regarding the study sequence to ensure the participants were adequately blinded.
After clicking on the start button, participants were asked a series of questions designed to gather demographic-related data. Adaptive questioning was used in this section to avoid displaying irrelevant questions. Participants were then asked a series of general questions related to SARS-CoV-2 and the COVID-19 pandemic.
The control group was then shown the prehospital COVID-19 guidelines. They were then asked to answer a second set of questions before being prompted to evaluate the learning path up to this point. Only then could they access the e-learning module and download their certificate.
The e-learning group followed the same path at first, but accessed the e-learning module immediately after being shown the guidelines. This group then completed the same second set of questions, was asked to evaluate the learning path (which in this case included the e-learning module), and was finally allowed to download the completion certificate.
The main outcome was defined as the difference in the proportion of correct choice of PPE before and after the course, assessed by means of short clinical scenarios.
Secondary outcomes were stratification of the main outcome by profession and by personal history of COVID-19 (whether or not the participant had been infected), accuracy of donning and doffing sequences reconstitutions, differences in the rates of overuse and underuse of PPE, confidence in one’s ability to use PPE, perceived usefulness of the course, and satisfaction regarding the course. The latter three outcomes were assessed by means of a 5-point Likert scale.
Data was electronically recorded and securely stored in an encrypted MariaDB database (Version 5.5.5; MariaDB Foundation). At the end of the study, all data was extracted to CSV format by the only investigator who had access (L Suppan). No personally identifiable data (including name, date of birth, email, or IP address) was ever asked for or recorded.
The extracted data were imported into Stata (StataCorp LLC). Variables were renamed to facilitate their understanding by the blinded data analysts. All data that could have enabled data analysts to identify group allocation were removed, and data fields were sorted accordingly. Incomplete questionnaires were excluded at this stage. The control and e-learning groups were renamed using city names (Moscow and Nairobi), and all relevant data were exported by L Suppan to a Stata .dta file and sent to L Stuby and MA for analysis.
As guidelines differ from region to region, we decided to only include prehospital staff working in the Geneva emergency medical system, thereby using a convenience sample rather than performing a sample size calculation.
Though most outcomes were electronically recorded and their interpretation therefore was generally independent of subjective human evaluation, comments had to be assessed by outcome assessors. Two assessors (L Stuby and PC), blinded as to group allocation, were asked to independently assess all comments. The nature of comments were to be rated as “positive,” “negative,” or “neutral” regarding the study, and as to whether they challenged Infection Prevention and Control (IPC) guidelines (binary, yes versus no). Disagreements were solved by sending the unclear comments to a third outcome assessor (BG), who was blinded to the previous assessments.
Statistical analysis was performed using Stata 15. Continuous independent outcomes were assessed using the Student
Continuous paired data were assessed using either the paired Student
Stratification was defined a priori based on expertise level (EMTs, paramedics, or physicians) and COVID-19 status (negative; confirmed, quarantined; confirmed, cured or unknown). Two post hoc sensitivity analyses were conducted.
The original data has been deposited to Mendeley Data [
Of the 291 randomized participants, 176 (60.5%) completed the trial (
Study flowchart.
The blinded data analysts (L Stuby and MA) excluded two surveys from the e-learning group as the participants’ professions did not match the target population (one “ambulance driver” and one “other”). They also excluded one survey from the control group because of incoherent answers (
Participant characteristicsa.
Characteristics | Control (n=85) | E-learning (n=88) | |
|
|||
|
Student paramedic | 5 (5.9) | 10 (11.4) |
|
Emergency medical technician | 11 (12.9) | 12 (13.6) |
|
Paramedic | 61 (71.8) | 60 (68.2) |
|
Physician | 8 (9.4) | 6 (6.8) |
Gender, female, n (%) | 32 (37.6) | 28 (31.8) | |
Age (years), median (Q1-Q3) | 35 (30-42) | 34 (28-40) | |
Professional experience (years), median (Q1-Q3) | 9 (3-15) | 7 (2-12) | |
|
|||
|
No/does not remember | 73 (85.1) | 70 (79.6) |
|
Yes | 12 (14.1) | 18 (20.4) |
Coronavirus disease status, positive, n (%) | 7 (8.3) | 6 (6.8) | |
Local guideline seen, yes, n (%) | 79 (92.9) | 84 (95.4) | |
Last time guideline seen (days), median (Q1-Q3) | 5 (3-10) | 5 (2-8) | |
Specific coronavirus disease course followed, yes, n (%) | 28 (32.9) | 32 (36.4) |
aTotals may not equal to 100% due to rounding.
There was no significant difference in baseline knowledge between groups. Though the baseline proportion of adequate PPE choice was high (75%, IQR 50%-75%), description of the donning sequence (assessed preintervention) was in most cases incorrect, as only 7 (4%) of the participants were able to reconstitute it accurately, with a similar proportion between groups. The donning sequence initially displayed in the survey was left unchanged by 7% of the participants (12/173, 6 per group).
Adequate choice of PPE was significantly increased in both groups after the intervention (
Change in proportion (%) of adequate choice of personal protective equipment.
Adequate choice of personal protective equipment (main outcome).
Outcomes | Control (n=85) | E-learning (n=88) | ||
Main outcome: difference in percentage of correct answers, median (Q1-Q3) | 8 (8 to 33) | 17 (8 to 33) | .27 | |
|
||||
|
Paramedic | 8 (0 to 25) | 17 (8 to 33) | .15 |
|
Paramedic student | 25 (8 to 33) | 29 (25 to 33) | .49 |
|
Emergency medical technician | 25 (8 to 33) | 8 (–8 to 33) | .88 |
|
Physician | 25 (13 to 38) | 13 (–8 to 42) | .30 |
|
||||
|
Negative for coronavirus disease | 8 (8 to 33) | 25 (8 to 33) | .20 |
|
Positive for coronavirus disease | 8 (–8 to 33) | 0 (–8 to 17) | .37 |
Confidence in the ability of using PPE was identical before and after the course in the e-learning group (
While most participants found the course useful (68.5%, 95% CI 61.5%-75.3%), the proportion of participants finding the course “very useful” was significantly higher (
Participant satisfaction.
The proportion of positive comments was similar between groups (14%, 95% CI 7%-21% in the e-learning group versus 19%, 95% CI 11%-27%;
Two post hoc sensitivity analyses were performed. The first was achieved by excluding all participants who answered they were either “very unsatisfied” or “unsatisfied” with the course. The second was performed by excluding participants whose compliance regarding COVID-19–specific IPC guidelines could be doubted; this included those who did not answer “systematic wearing of a double pair of gloves” to the question “Which of these measures is NOT one of the infection prevention measures?” (40 [45%] in the e-learning group versus 35 [41%]). These analyses did not demonstrate a significant change regarding the main outcome (
In this study, the proportion of those making an adequate choice regarding PPE increased significantly after prehospital personnel followed either web-based study path. Though the median of the increase was twice that of the e-learning group, this difference was not statistically significant.
Failure to reach significance might be explained by many different factors. The high baseline proportion of adequate choice of PPE in both groups may have dampened any relative impact of the intervention as there was little room left for improvement. Although this result might seem counterintuitive given the high rate of participants who responded that they did not attend (or did not recall attending) an IPC course prior to this study, more than 90% declared they had consulted the local guidelines recently. Most participants were therefore aware of the local recommendations, and the anxiety generated by the COVID-19 pandemic probably acted as a catalyst regarding their interest in such guidelines [
Another factor that might partly explain the lack of a significant difference is the sample size, which limited the power to detect small differences. Nevertheless, as PPE guidelines vary not only between countries, but also between Swiss cantons, it might have been inadequate to draw conclusions from a pooled group of paramedics working in different cantons with different guidelines [
Though necessary to assess participants’ knowledge and attitudes regarding PPE prior to the interventions, the first set of questions, along with the study title, might have acted as a primer and focused the participants’ attention on the specific contents that would be tested postintervention [
The relatively low level of satisfaction displayed by the participants should also be taken into account. Though e-learning modules and serious games usually increase participant satisfaction when compared to more traditional methods [
Though confidence in the ability to use PPE was maintained in the e-learning group, it significantly decreased after reviewing the guidelines in the control group. Participants who initially felt confident in their knowledge might have felt it was challenged after being asked specific questions [
Though incorrect answers regarding the donning sequence are easily understandable as they might primarily result from a lack of knowledge, an inability to correctly rebuild the doffing sequence is questioning. No less than four different broad categories of causes could be involved: ineffectiveness of the teaching material, inadequate sequence, flawed means of assessing the participants’ knowledge, or disagreement with the procedure outlined in the guidelines and in the e-learning module. As this result was unforeseen, the method used in this study was unfortunately ill-suited to the evaluation of the underlying causes. Nevertheless, with less than 10% of participants having left the initially displayed sequences unchanged, a technical flaw can reasonably be ruled out.
Apart from the abovementioned limitations, the ever-increasing knowledge regarding SARS-CoV-2 and COVID-19 might render both the guideline and the gamified e-learning module used in this study obsolete. However, current technological tools might mitigate this effect as they allow for a quick adaptation, even of highly interactive content [
Another limitation is the relatively small number of questions asked pre- and postintervention. Keeping the total number of questions and the time required to complete either study path relatively low was necessary to limit attrition [
Despite its limitations, this study also has some strengths, among which the quadruple-blind design and the relatively high response rate should be acknowledged. Moreover, as all answers were electronically recorded, there was no risk of an outcome assessment bias. Finally, as neither the control nor the e-learning path requires the physical presence of either participants or instructors, the framework used in this study could serve as the building ground for courses in the context of an epidemic or a pandemic such as the current COVID-19 situation.
The larger impact such a web-based study might have had, regardless of the effect of a specific intervention such as the gamified e-learning module, should be assessed, as a possible change in PPE consumption or infection rate among prehospital providers could ensue.
The potential impact of this gamified e-learning module on less experienced and less primed participants should also be evaluated to confirm (or refute) the theories outlined in this discussion. The gamified e-learning module can thus be freely downloaded from the study website in both a web (HTML5 with Flash fallback) and SCORM (Shareable Content Object Reference Model) format.
Among prehospital personnel with an already relatively high knowledge and experience regarding PPE use, both web-based study paths increased the rate of adequate choice of PPE. There was no major added value of the gamified e-learning module apart from preserving participants' confidence in their ability to correctly use PPE.
First email sent to participants belonging to the control group.
First email sent to participants belonging to the e-learning group.
Details of the answer set excluded by the blinded data analysts.
Checklist for Reporting Results of Internet E-Surveys
Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth
coronavirus disease
electronic learning
emergency medical technician
infection prevention and control
personal protective equipment
severe acute respiratory syndrome coronavirus 2
Shareable Content Object Reference Model
Service Mobile d’Urgences et de Réanimation
The authors would like to thank Mr Michel Hofer for his support.
Most of the authors participated in the development of the gamified e-learning module tested in this trial. Nevertheless, as this module is freely available, the authors deny any financial conflict of interest.
This randomized study was not prospectively registered. As the purpose of the study was to examine the effect of two different study paths only on providers’ knowledge and attitude toward PPE, registration of the trial was not performed as it is not deemed necessary by the International Committee of Medical Journal Editors. The editor is therefore following the guidelines suggested by the ICMJE and not mandating prospective registration of this randomized trial. However, readers are advised to carefully assess the validity of any potential explicit or implicit claims related to primary outcomes or effectiveness, as retrospective registration does not prevent authors from changing their outcome measures retrospectively.