This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The COVID-19 pandemic has considerably altered the regular medical education curriculum while increasing the need for health care professionals. Senior medical students are being incrementally deployed to the front line to address the shortage of certified physicians. These students, some of whom will be fast-tracked as physicians, may lack knowledge regarding the initial management of time-critical emergencies such as stroke.
Our aim was to determine whether an e-learning module could improve asynchronous distance knowledge acquisition of the National Institutes of Health Stroke Scale (NIHSS) in senior medical students compared to the traditional didactic video.
A randomized, data analyst–blinded web-based trial was conducted at the University of Geneva Faculty of Medicine between April and June 2020. Fifth year medical students followed a distance learning path designed to teach the NIHSS. The control group followed the traditional didactic video created by Patrick Lyden, while the e-learning group followed the updated version of a previously tested, highly interactive e-learning module. The main outcome was the score on a 50-question quiz displayed upon completion of the learning material. The difference in the proportion of correct answers for each specific NIHSS item was also assessed.
Out of 158 potential participants, 88 started their allocated learning path and 75 completed the trial. Participants who followed the e-learning module performed better than those who followed the video (38 correct answers, 95% CI 37-39, vs 35 correct answers, 95% CI 34-36,
Compared to the traditional didactic video, a highly interactive e-learning module enhances asynchronous distance learning and NIHSS knowledge acquisition in senior medical students.
The swift strengthening of public health policies in the context of the COVID-19 crisis has wrought deep changes in the regular medical education curricula of many countries [
Traditional classroom or bedside teaching can be difficult to conduct in certain situations, such as a pandemic [
Since the release of Patrick Lyden’s didactic video in 1994 [
This first study was performed with the participants present at the study site; therefore, they could immediately access technical support if needed. Moreover, although most results favored the use of the e-learning module, the control group was better at scoring the ataxia element than the e-learning group. Although video extracts were used within the e-learning module to demonstrate the assessment of almost all NIHSS items, the chapter regarding the ataxia element did not contain any video extracts. We therefore hypothesized that systematically embedding videos could improve NIHSS learning acquisition, and we updated the module accordingly.
Given the need for social distancing during the COVID-19 pandemic, our goal was to compare medical students’ asynchronous distance learning of the NIHSS using two different teaching tools: the gold standard didactic video and the updated version of our e-learning module.
We performed a randomized, controlled, data analyst–blinded, web-based trial following the CONSORT-EHEALTH guidelines and incorporating relevant elements from the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) checklist [
Because the study included no patients and as no health outcomes were recorded, trial registration was not required according to the International Committee of Medical Journal Editors guidelines. Although the participants were not part of a vulnerable group according to Swiss federal law on human research [
After gathering the necessary authorizations, the UGFM students’ secretary transmitted the exact number of fifth year medical students to MS, who performed a 1:1 computer assisted randomization without having access to any other data regarding the students. MS then created specific identifiers that were transmitted back to the UGFM along with a mailing template
The students were informed of the goals of the investigation, were given information regarding data security and anonymization procedures, and were supplied with the email addresses of three investigators to allow them to ask further questions. Students who elected to browse the website were provided with additional information as well as with a link to a full 4-page consent form in PDF format that they could either print or save. Using their identifiers to log into the site was considered as acceptance to participate in the study. All participants were free to withdraw at any time. No financial incentive was provided.
We created a specific web-based platform under the Joomla 3.9 content management system (Open Source Matters) [
The e-learning module contains 16 independent chapters. The first chapter is the introduction, which is automatically displayed when the module is launched. Prevention of content skipping is the first learning mechanic used in the module, and it already appears in the introductory slides (
A table of contents is displayed as soon as the user has completed the introduction. The user can then choose to review the introduction or to access any other chapter apart from the summary (
Prevention of content skipping. The user cannot click on the
The user has clicked on one of the two buttons, and the learning content is now displayed in a lightbox. The
Both buttons have been clicked, and the user has seen both lightboxed slides. The
Table of contents of the e-learning module. The
Thirteen chapters are used to explain each specific NIHSS item (3 chapters are used to cover the first item, “consciousness,” which is divided into 3 elements). Although the chapters are numbered consecutively according to the NIHSS scoring logic, the user can freely elect the order in which to follow the chapters.
All chapters include at least two learning mechanics. First, each chapter begins by displaying the NIHSS score specific to the scoring item, and users are once again prevented from skipping content, as they must click on each numbered button to discover the score (
The second learning mechanic is linked to the use of subtitled videos. Video extracts are shown to the user, who must correctly score the NIHSS item (
Feedback is provided for each question [
If the answer was correct, feedback is also given to reinforce the message (
Specific interactions were designed to further illustrate particular elements, such as visual field deficits (
Prevention of content skipping. The user cannot click on the
Video-based question. The user must choose the correct score for the patient displayed in the video before clicking on
Wrong answer feedback and clue. The user can choose either to try again (
Positive feedback after a correct answer.
Interactive explanation of visual field deficit.
Animation used to explain extinction and inattention.
The user can choose to exit the module at any time, as a prompt will be shown to allow the user to either resume the module or reset it. Before the summary chapter can be activated, users must complete a 14th chapter, which details the “coma score.”
This e-learning module, along with its previously studied iteration, can be accessed freely on the internet [
Immediately after login, the medical students learned which group they had been allocated to and were asked a first set of 6 questions displayed over a single page. Upon answering these questions, they could then access the learning material. No time limit was applied apart from the study end date (June 8, 2020). Once the learning material was completed, students were allowed to proceed to a 50-question quiz. This quiz was identical for all participants and contained five questions related to basic NIHSS concepts, followed by the clinical evaluation of 3 patients taken from Patrick Lyden’s certification videos. The NIHSS elements were displayed and scored in sequence according to the NIHSS scoring logic. After finalizing the quiz, participants were given their overall score as well as the possibility to review all questions at will and were shown their answers along with the correct answers. Then, 4 questions, based on a 5-point Likert scale, were asked to assess secondary outcomes, such as satisfaction. Students were finally given access to both the video and the e-learning module to discover the other teaching modality and/or to review the one they had just followed.
The primary outcome of the study was the proportion of correct answers to the 50-question quiz. Secondary outcomes were the proportion of correct answers for each specific NIHSS item, user satisfaction, perceived adequacy of the time needed to complete the course, perceived difficulty of the course, probability that the participant would recommend the course, and whether the learning path had been completed over multiple days.
Data were securely stored on an encrypted MariaDB 5.5.5 database (MariaDB Foundation) located on a Swiss server before being extracted in comma-separated values (CSV) format. We used STATA (StataCorp) for data curation and anonymization.
STATA 15.1 was used by L Stuby for statistical analysis. Incomplete answers to the 50-question quiz were not analyzed.
Normality was assessed by graphical evaluation and, if in doubt, we used the Shapiro-Wilk test. We applied the Fisher exact test to categorical variables and the Student
We used a convenience sample and calculated the power post hoc. We defined 4 sensitivity analyses a priori according to whether the participant had prior knowledge of the NIHSS, had already followed a specific NIHSS course, had worked in either an intensive care unit or in a neurology or neurosurgery ward for more than 3 months, or had completed the learning path over multiple days. This was defined as more than 12 hours elapsed between initiation and completion of the course.
Finally, we performed univariate followed by multivariable linear regression to look for possible confounding factors.
Our curated data file is available on Mendeley Data [
Out of 158 potential participants, 75 (47.5%) completed the trial (
After the first mailing (April 28, 2020), 21 students completed the trial. The first reminder (May 11, 2020) led 29 more students to complete the course, while another 25 participated after the second and last reminder (May 18, 2020).
Study flowchart.
Participant data (N=75).
Characteristic | Value | |
|
Video group (n=34) | E-Learning group (n=41) |
Age, median (Q1-Q3) | 24 (23-25) | 24 (23-24) |
Prior knowledge of NIHSSa application, n (%) | 3 (9) | 3 (7) |
Specific NIHSS course followed, n (%) |
4 (12) |
2 (5) |
E-learning NIHSS course followed, n (%) | 0 (0) | 0 (0) |
Had worked in intensive care unit or neurology ward, n (%) | 0 (0) | 1 (2) |
Missing data, n (%) | 2 (6) | 2 (5) |
aNIHSS: National Institutes of Health Stroke Scale.
Participants who followed the e-learning module performed better than those who followed the video (38 correct answers, 95% CI 37-39, vs 35, 95% CI 34-36,
The rate of “very satisfied” participants was higher in the e-learning group (14/35, 40%; 95% CI 24%-56%) versus the video group (4/26, 15%; 95% CI 5%-25%,
Quiz results.
Item | Video group (n=34) | E-Learning group (n=41) | ||
Overall score, mean (SD) | 35 (3) | 38 (3) | <.001 | |
Overall score, 95% CI | 34-36 | 37-39 | N/Aa | |
|
||||
|
Key NIHSSb concepts | 5 (4-5) | 5 (5-5) |
.02 |
|
Consciousness – Global | 2 (2-2) | 3 (2-3) | <.001 |
|
Consciousness – Questions | 3 (2-3) | 3 (3-3) | .70 |
|
Consciousness – Commands | 2 (2-3) | 3 (2-3) | .06 |
|
Gaze | 2 (2-3) | 3 (2-3) | .34 |
|
Visual | 2 (2-2) | 2 (2-2) | .23 |
|
Facial Palsy | 1 (0-2) | 2 (1-2) | .04 |
|
Motor arm | 4 (4-5) | 5 (4-5) | .17 |
|
Motor leg | 5 (4-6) | 5 (4-5) | .23 |
|
Ataxia | 1 (1-1) | 1 (1-2) | .03 |
|
Sensory | 3 (2-3) | 3 (3-3) | .04 |
|
Language | 1 (1-2) | 1 (1-1) | .63 |
|
Dysarthria | 2 (2-2) | 2 (2-2) | .07 |
|
Extinction and inattention | 2 (2-3) | 2 (2-3) | .14 |
aN/A: not applicable.
bNIHSS: National Institutes of Health Stroke Scale.
Satisfaction of the participants in the e-learning and video groups regarding the learning method.
Although the precise total learning time dedicated to either method could not be assessed due to the study design, no statistical difference regarding the perceived duration of the course was identified (80% (28/35) adequate in the e-learning group, 95% CI 67%-93%, vs 65% (17/26) 95% CI 47%-83%,
Likelihood that the participants in the e-learning and video groups would recommend the course.
The post hoc calculation showed a power of 97%. None of the 4 preplanned sensitivity analyses showed any major changes in the direction of the effect. The multivariable linear regression only showed a minor change in the coefficient (<15%), confirming these results.
In this study, asynchronous distance learning using a highly interactive e-learning module yielded better results than following the traditional didactic video on the web. The superiority of a previous version of this module has already been established in Swiss paramedics following an onsite computer-based course [
The shortcomings we had identified in the previous iteration of the e-learning module seem to have been addressed, as embedding cutscenes from the original video into every chapter of the module improved the impact of the module on knowledge acquisition. The use of short videos associated with active learning activities such as guiding questions or interactive elements has been shown to enhance knowledge acquisition and retention [
Slightly less than half of all potential participants completed their allocated learning path. Considering that the learning material was optional and that students’ summative assessments of this semester were replaced by formative assessments, the participation rate is rather encouraging given the global lack of incentive. More encouraging still is the proportion of students who would recommend the course to their peers, as such mechanisms may increase students' involvement [
The quiz shown to the participants upon completion of the learning material included not only the full evaluation of 3 different stroke patients, but also 5 general questions we had designed
In many hospitals, the NIHSS is commonly used to triage stroke victims and help reduce both door-to-CT (CT: computed tomography) and door-to-needle times [
This study has limitations that must be acknowledged. The main limitation is that we only measured immediate knowledge acquisition; we were unable to assess knowledge retention due to the study design and the limited timeframe. As this latter parameter is critical to the clinical application of the NIHSS, further studies will be needed to assess whether the e-learning method improves retention and leads to more accurate application of the scale. Moreover, the precise time taken to complete either learning method was not evaluated in this study. While it can be argued that dedicating more time to learning given content should yield better results, studies have shown that engagement is the most important factor regarding knowledge acquisition [
Despite these limitations, this study also has several strengths, including the randomization, the blinding mechanisms, the electronic data acquisition, the originality of the learning method and its mode of delivery in the context of the COVID-19 pandemic.
Compared to the traditional didactic video, a highly interactive e-learning module enhances distant NIHSS knowledge acquisition in medical students.
Mailing template.
Original questions used in the 50-question quiz.
CONSORT-eHEALTH checklist (V 1.6.1).
Checklist for Reporting Results of Internet E-Surveys
comma-separated values
computed tomography
National Institutes of Health Stroke Scale
University of Geneva Faculty of Medicine
The authors would like to thank Mr Daniel Scherly from the UGFM for his technical assistance as well as for dispatching the emails and reminders on our behalf.
None declared.
This randomized study was not registered. The authors explained that their study included no patients and no health outcomes were recorded. According to the ICMJE rules, if the purpose is to examine the effect only on the providers rather than patients, then registration is not necessary. However, readers are advised to carefully assess the validity of any potential explicit or implicit claims related to primary outcomes or effectiveness.