This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Traditional approaches to health professional education are being challenged by increased clinical demands and decreased available time. Web-based e-learning tools offer a convenient and effective method of delivering education, particularly across multiple health care facilities. The effectiveness of this model for health professional education needs to be explored in context.
The study aimed to (1) determine health professionals’ experience and knowledge of clinical use of vancomycin, an antibiotic used for treatment of serious infections caused by methicillin-resistant
We conducted a study on the design and implementation of a video-enhanced, Web-based e-learning tool between April 2014 and January 2016. A Web-based survey was developed to determine prior experience and knowledge of vancomycin use among nurses, doctors, and pharmacists. The Vancomycin Interactive (VI) involved a series of video clips interspersed with question and answer scenarios, where a correct response allowed for progression. Dramatic tension and humor were used as tools to engage users. Health professionals’ knowledge of clinical vancomycin use was obtained from website data; qualitative participant feedback was also collected.
From the 577 knowledge survey responses, pharmacists (n=70) answered the greatest number of questions correctly (median score 4/5), followed by doctors (n=271; 3/5) and nurses (n=236; 2/5;
A novel Web-based e-learning tool was successfully developed combining game design principles and humor to improve user engagement. Knowledge gaps were identified that allowed for targeting of future education strategies. The VI provides an innovative model for delivering Web-based education to busy health professionals in different locations.
The development of Internet-based learning (IBL) for health care professionals has increased in recent years [
Serious games have been defined as “interactive computer applications, with or without significant hardware components” that are designed to entertain while achieving changes in knowledge or skills [
Use of serious game methodologies to deliver health professional education has been reported in previous studies [
Most e-learning tools in health care have targeted specific groups, such as medical or nursing students, physicians, or nurses [
The aims of this study were to (1) report the design and implementation of a Web-based, interactive e-learning tool providing education on the dosing, administration, and TDM of vancomycin, (2) assess health professionals’ preintervention knowledge of vancomycin use in order to inform development of the e-learning tool, and (3) assess health professionals’ initial acceptance of the VI.
This prospective design and implementation study of a video-enhanced, Web-based e-learning tool took place in Illawarra Shoalhaven Local Health District (ISLHD) and South Eastern Sydney Local Health District (SESLHD), located in New South Wales (NSW), Australia. These health districts cover a geographic area of 6331 km2 and have an estimated population of 1.17 million, reaching from central Sydney to 3 h drive south [
Timeline of Vancomycin Interactive design, implementation, and evaluation. ISLHD: Illawarra Shoalhaven Local Health District; SESLHD: South Eastern Sydney Local Health District.
An anonymous open Web-based survey was created using SurveyMonkey (SurveyMonkey Inc, Palo Alto, CA) to determine confidence, experience, and knowledge of vancomycin before the VI. The survey was developed locally by the antimicrobial stewardship and educator pharmacists as part of routine activities, with input from the infectious diseases team. Clinical content was based on the Australian Therapeutic Guidelines: Antibiotic, Version 15, 2014 [
Similar to the survey, clinical content of the VI was developed locally, based on the Therapeutic Guidelines: Antibiotic [
A single interactive video was produced due to financial constraints; there was the expectation that all professional groups should have rudimentary knowledge of clinical vancomycin use. The VI (ISLHD) was hosted on an open website [
The user interface consisted of video clips interspersed with interactive question and answer scenarios placed at the specific points, so that technical content felt organic to the narrative (
Quotes for production were obtained from three developers in accordance with NSW Health policy, with financial support provided internally by the Clinical Governance Unit of the health district. Content development began in April 2014, and the video was filmed using professional actors in November 2014. Postproduction modifications were made to the video until release in July 2015. In early 2015, the website was established to promote improved access to the VI, and to include additional clinical content not contained in the VI. Testing of content and usability was performed by pharmacists and infectious diseases doctors (N=8) at the study site, with feedback provided by email to the study investigators. Feedback from testers predominantly related to accuracy of the clinical content in the context of the narrative, and informed the final iteration of the VI. The first phase of dissemination and advertisement (email, newsletters, link on intranet home page) to ISLHD staff occurred on July 27, 2015 (
Filming of the Vancomycin Interactive (VI).
Dramatic tension created the basis for the Vancomycin Interactive’s plot.
Example of user interface for an interactive question from the Vancomycin Interactive (VI).
Following release of the VI, qualitative survey responses were assessed to inform the investigators about user acceptability and suggestions for improvement. The qualitative survey was open between December 1, 2015 and January 31, 2016, in order to conclude before the annual intake of new junior doctors in February 2016 (
The primary outcome measure was comparative vancomycin knowledge between health professions and self-reported levels of confidence and experience. Vancomycin knowledge responses from website data (not linked at a participant level) were also assessed and compared with the knowledge survey. In addition, qualitative feedback on the VI was evaluated using a 5-point Likert scale and free text responses that were grouped into key themes. Assessments were derived from survey responses and VI website data. Technical issues around compatibility with desktop and mobile operating systems and Web browsers were also assessed. Reporting of outcomes on quantitative postintervention survey data, clinical measures of quality vancomycin use such as therapeutic vancomycin plasma levels, and clinical outcomes related to vancomycin treatment was beyond the scope of this study.
Chi-square and Fisher exact tests were used for proportions. Chi-square for trend was used to determine trend between professions for knowledge questions. Kruskal-Wallis and Mann-Whitney
Ethics approval was granted by the Joint University of Wollongong and Illawarra Shoalhaven Local Health District Health and Medical Human Research Ethics Committee (EC00150; approval number HE15/005). The VI website contained a disclaimer that anonymous data collected from the video could be used for research purposes.
The response rate to the survey was 26.87% (577 responses from 2147 email recipients). The response rates by profession were 24.4% (236/967) for nurses, 25.33% (271/1070) for doctors, and 63.6% (70/110) for pharmacists (
As shown in
Number of correct responses to Web-based vancomycin knowledge survey.
Survey question | Nurse |
Doctor |
Pharmacist |
Total |
|
Loading dose | 46 (19.5) | 112 (41.3) | 59 (84) | <.001 | 217 (37.6) |
Maintenance dose | 58 (24.6) | 126 (46.5) | 58 (83) | <.001 | 242 (41.9) |
Administration rate | 152 (64.4) | 160 (59.0) | 62 (89) | <.001 | 374 (64.8) |
First level timing | 70 (29.7) | 155 (57.2) | 59 (84) | <.001 | 284 (49.2) |
Target trough range | 136 (57.6) | 234 (86.3) | 65 (93) | <.001 | 435 (75.4) |
Median total score (IQRa) | 2 (1-3) | 3 (3-4) | 4 (3-4) | <.001 | 3 (2-4) |
aIQR: interquartile range.
Multivariate analysis (
Multivariate analysis of vancomycin knowledge survey responses (N=577).
Topic | Profession | Experience or confidencea | |||||||
Nurse |
Doctor | Pharmacist | 1 |
2 | 3 | 4 | |||
ORc |
- | 2.6 |
16.8 |
- | 1.4 |
4.6 |
11.1 |
||
- | <.001 | <.001 | - | .25 | <.001 | <.001 | |||
OR |
- | 2.5 |
12.1 |
- | 1.0 |
2.2 |
3.3 |
||
- | <.001 | <.001 | - | .85 | .006 | .03 | |||
OR |
- | 1.0 |
2.9 |
- | 2.1 |
4.7 |
5.7 |
||
- | .82 | .01 | - | .001 | <.001 | <.001 | |||
OR |
- | 2.7 |
8.5 |
- | 3.6 |
6.8 |
4.0 |
||
- | <.001 | <.001 | - | <.001 | <.001 | .008 | |||
OR |
- | 3.9 |
5.6 |
- | 3.4 |
5.7 |
5.7 |
||
- | <.001 | .001 | - | <.001 | <.001 | .03 | |||
- | .8 |
1.7 |
- | .9j |
|||||
- | <.001 | <.001 | - | <.001 |
aExperience or confidence: 1, none; 2, a little; 3, moderate; 4, very experienced or confident.
bref: reference group for multivariate analysis.
cOR: odds ratio.
dmaint dose: maintenance dose.
eadmin rate: administration rate.
flevel timing: timing of first level.
gtrough range: target range for plasma trough level.
htotal correct: nurse (all levels), doctor (all levels), pharmacist (all levels).
i
jAverage of responses to three vancomycin experience or confidence questions, therefore a multiple regression was performed for total correct.
Subsequent mediation analysis revealed that vancomycin experience significantly mediated the effect of profession on total score (total indirect effect:
Responses to the VI were analyzed using background website data received from July 27, 2015 to November 14, 2015, with ISLHD as the target population group. The initial dropdown question asking the user’s profession was answered by 389 participants, of which 163 health professionals (41.9% of those answering the initial profession question) completed all 10 questions (
Number (%) of correct answers on first attempt by nurses, doctors, and pharmacists from VI data.
Questiona | Nurse |
Doctor |
Pharmacist |
Total |
|
1. Loading dose | 19 (30) | 36 (42) | 9 (64) | .05 | 64 (39) |
2. Maintenance dose | 50 (79) | 59 (69) | 11 (79) | .32 | 120 (74) |
3. Compatible fluids | 53 (84) | 67 (78) | 11 (79) | .76 | 131 (80) |
4. Administration rate | 56 (89) | 55 (64) | 14 (100) | <.001 | 125 (77) |
5. Timing of first level | 20 (32) | 49 (57) | 12 (86) | <.001 | 81 (50) |
6. Target trough level | 47 (75) | 72 (84) | 12 (86) | .34 | 131 (80) |
7. Level of 35 mg/L | 43 (68) | 68 (79) | 14 (100) | .02 | 125 (77) |
8. Level of 20 mg/L | 49 (78) | 81 (94) | 13 (93) | .01 | 143 (88) |
9. Level of 26 mg/L | 46 (73) | 55 (64) | 12 (86) | .20 | 113 (69) |
10. Subsequent levels | 27 (43) | 45 (52) | 7 (50) | .52 | 79 (48) |
Average score | 65% | 68% | 82% | <.001 | 68% |
aVI questions are shown in
b
The rates of correct response from the VI were significantly higher than the knowledge survey for maintenance dose (74% VI vs 42% survey;
Among the 163 VI participants, 51 (31%) responses were received. Responses were predominantly positive, as shown in
Qualitative responses (%) following participation in the Vancomycin Interactive (VI).
Survey statement or question | Strongly |
Agree | Neutral | Disagree | Strongly disagree |
11 (22) | 29 (57) | 8 (16) | 3 (6) | 0 (0) | |
Using the VI has improved my performance (n=50) | 8 (16) | 28 (56) | 12 (24) | 2 (4) | 0 (0) |
aVI: Vancomycin Interactive.
When users were asked, “What’s good about the VI in comparison to other e-learning modules?” 28 free text responses were received. It was found that 4 responses (14%) were related to not being able to load the video. Key themes from the remaining 24 responses (86%) were “entertaining,” “engaging,” “a lighter approach to learning,” “more real life,” and “held attention.” The question, “Does the training provided by the VI meet your needs? If not, what can be improved?” received 23 free text responses. A total of 16 respondents (70%) reported, “yes it met needs;” 2 users (10%) stated issues loading VI; 3 users (13%) requested printable resources; 1 user was “not sure;” and 1 user requested more information to be available when answering questions. All qualitative survey responses are provided in
We have reported on the design, implementation, and user evaluation of a novel Web-based e-learning tool for education of health professionals on clinical use of the antibiotic vancomycin. The VI was developed for noncommercial use and targeted three health professional groups across multiple hospital sites. Responses from the survey that preceded the VI demonstrated a global lack of knowledge on the safe and effective use of vancomycin among nurses and doctors, justifying a Web-based learning approach that was suitable for disparate geographical locations. Pharmacists were shown to be more knowledgeable on clinical vancomycin dosing, administration, and TDM.
As expected, self-reported levels of confidence and experience were correlated with increased likelihood of correct responses to the knowledge survey questions. Responses from the Web-based knowledge survey and VI data were only similar for three of the five common questions, loading dose, timing of first level, and the target trough level (see
Numerous studies have reported the development and evaluation of serious games for training health professionals, but few have targeted multiple professions [
Feedback from the majority of the participants suggested that the VI enhanced their vancomycin knowledge (79%) and improved their performance (72%). This supports the VI as a Web-based resource to provide health care professionals with training on clinical use of vancomycin. Qualitative responses were generally positive, further supporting the use of the VI for health professional education. The main challenges for implementation of the VI related to developing clinical content for the video that would remain applicable to all three professional groups, without creating a tool that would take too long to complete. Advertising the tool using different media was also challenging, as the tool was made available across two health districts with multiple hospitals, and the target professional groups may have preferred to receive alerts regarding content in different ways.
The creation of a brief, Web-based, entertaining educational tool was the purpose of the project, whereby no further mandatory training burden was placed on staff. As distinct from existing local mandatory learning modules, the VI was intended for use among clinical staff involved in vancomycin use. Employing serious game design concepts may provide greater educational benefit than traditional computer-based learning methods through the use of greater interactivity, entertainment, and scoring; however, further published comparisons are required [
We studied the logistics and design of a Web-based e-learning tool incorporating interactive video content for health professional education relating to clinical use of the antibiotic vancomycin. Postintervention knowledge and clinical outcomes were not reported here; these form the basis of ongoing research that will be reported separately. The use of an open Internet site allowed for potential diffusion worldwide, since users outside our organizations may have found the VI using an Internet search engine. In August 2015, the website was also shared on a professional network with members outside the targeted health district. As a result, there was some unintended use of the video before its general release. However, the greatest number of Web sessions was from ISLHD, and employees of the target ISLHD hospitals may not have been physically located in the region while completing the VI.
Question design within the VI was limited to multiple choice and multiple answer questions. Further variation in question types such as open questions, as previously reported [
There was relatively low uptake of the VI among clinical staff during the study period despite broad advertisement; this limited the statistical power of the study and highlighted the challenge of using a new e-learning tool for delivery of noncompulsory training material to health professionals. Reasons for this probably related to the following: (1) the VI was not mandatory learning, so health professionals who did not regularly use vancomycin may not have been motivated to participate; (2) competing education priorities in those health professionals not otherwise intrinsically motivated to participate; (3) lack of time out from clinical responsibilities; (4) the likelihood that multiple staff completed the VI together, meaning that the VI’s reach might have been greater than the results demonstrated; (5) the tool was not targeted toward a specific profession; and (6) not being able to access the VI using hospital computers, which may have hampered widespread use by health professionals during office hours. However, there were only four reports of the VI not loading from 51 survey responses, suggesting that the majority of participants could access the VI. Although the free access website allowed for participation during working hours, there may have been less motivation to perform work-related education in this setting. It was expected that the greatest amount of participation would occur during working hours on hospital computers. Clinical indications for vancomycin were not addressed by the VI, as its primary purpose was to improve knowledge once the decision to prescribe had been made.
Our study presented a model for adopting serious game concepts in combination with humor to develop and conduct Web-based health professional education in a light-hearted, interactive, and entertaining way. This model may be useful in settings where use of face-to-face education is limited by resources and geography. As the VI learning material was made available around the world, it showcased another significant benefit of open Web-based education resources. Health professionals and health care organizations with the same learning needs can reuse the material we have published rather than expending resources to develop similar material.
We demonstrated a novel Web-based e-learning tool that used humor and some game design principles to deliver health professional education on the commonly used antibiotic vancomycin. The VI was well accepted by users, and it was thus useful for delivering the intended health professional education. Future learning needs for different professional groups were identified through both the Web-based knowledge survey and VI data. This will allow tailoring of face-to-face education programs, in addition to subsequent versions of the VI that will embed robust gaming methodology. Further research will be aimed at measuring the effect on knowledge of the VI compared with a traditional email intervention and examining the impact of the VI on clinical vancomycin use.
Vancomycin Web-based knowledge survey questions.
Vancomycin Interactive questions.
Qualitative survey feedback on the Vancomycin Interactive.
Internet-based learning
interquartile range
Illawarra Shoalhaven Local Health District
methicillin-resistant Staphylococcus aureus
New South Wales
South Eastern Sydney Local Health District
therapeutic drug monitoring
Vancomycin Interactive
The authors would like to thank Michael Boland from Digital League and team for production and programming of the VI and its website. We also thank the pharmacy and infectious diseases teams at ISLHD and SESLHD for user testing of the survey and VI. No financial support was reported. The filming and production of the VI was funded internally by Illawarra Shoalhaven Local Health District. Preliminary results from this study were presented in abstract form at Medicines Management 2015: the 41st Society of Hospital Pharmacists of Australia National Conference, Melbourne, Australia, December 3-6, 2015.
None declared.