Web-Based Educational Intervention to Improve Knowledge of Systematic Reviews Among Health Science Professionals: Randomized Controlled Trial

Background Lack of knowledge of systematic reviews (SRs) could prevent individual health care professionals from using SRs as a source of information in their clinical practice or discourage them from participating in such research. Objective In this randomized controlled trial, we evaluated the effect of a short web-based educational intervention on short-term knowledge of SRs. Methods Eligible participants were 871 Master’s students of university health sciences studies in Croatia; 589 (67.6%) students who agreed to participate in the trial were randomized using a computer program into 2 groups. Intervention group A (294/589, 49.9%) received a short web-based educational intervention about SR methodology, and intervention group B (295/589, 50.1%) was presented with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist. The participants’ knowledge of SRs was assessed before and after the intervention. The participants could not be blinded because of the nature of the intervention. The primary outcome was the difference in the percentage of correct answers about SR methodology per participant between the groups after the intervention, expressed as relative risk and 95% CI. Results Results from 162 and 165 participants in the educational intervention and PRISMA checklist groups, respectively, were available for analysis. Most of them (educational intervention group: 130/162, 80.2%; PRISMA checklist group: 131/165, 79.4%) were employed as health care professionals in addition to being health sciences students. After the intervention, the educational intervention group had 23% (relative risk percentage) more correct answers in the postintervention questionnaire than the PRISMA checklist group (relative risk=1.23, 95% CI 1.17-1.29). Conclusions A short web-based educational intervention about SRs is an effective tool for short-term improvement of knowledge of SRs among health care studies students, most of whom were also employed as health care professionals. Further studies are needed to explore the long-term effects of the tested education. Trial Registration OSF Registries 10.17605/OSF.IO/RYMVC; https://osf.io/rymvc


2a-i) Problem and the type of system/solution
"Knowledge of SRs in trainees can help not only in developing useful skills in critical appraisal but also in addressing important clinical questions and serve as a strong basis to design new, original research studies that will fill the gaps and answer relevant and unsolved clinical questions." 2a-ii) Scientific background, rationale: What is known about the (type of) system " It has been shown that better knowledge and more positive attitudes toward EBM among medical students are associated with the exposure to the vertical subject on research in biomedicine and activities of The Cochrane Collaboration." Does your paper address CONSORT subitem 2b?
"In this randomized controlled trial (RCT), we evaluated the effect of a short web-based educational intervention about SRs on short-term knowledge of SR among students of health sciences studies in Croatia." METHODS 3a) CONSORT: Description of trial design (such as parallel, factorial) including allocation ratio "We conducted an RCT with 2 parallel groups and 1:1 participant allocation." 3b) CONSORT: Important changes to methods after trial commencement (such as eligibility criteria), with reasons There were no important changes to methods after trial commencement. 3b-i) Bug fixes, Downtimes, Content Changes 4a) CONSORT: Eligibility criteria for participants "The participants were students of Master's university health sciences studies in Croatia." 4a-i) Computer / Internet literacy 4a-ii) Open vs. closed, web-based vs. face-to-face assessments: "Students from eligible institutions were contacted via email by coauthors (MC, MN, KI, DA, NS, SZ, and SM) employed in these institutions and invited to participate in the study on brief web-based education about SRs of the literature." 4a-iii) Information giving during recruitment 4b) CONSORT: Settings and locations where the data were collected "There were 8 eligible institutions in Croatia for this study, and we invited all of them." 4b-i) Report if outcomes were (self-)assessed through online questionnaires Outcomes were (self-)assessed through online questionnaires. 4b-ii) Report how institutional affiliations are displayed 5) CONSORT: Describe the interventions for each group with sufficient details to allow replication, including how and when they were actually administered 5-i) Mention names, credential, affiliations of the developers, sponsors, and owners 5-ii) Describe the history/development process 5-iii) Revisions and updating 5-iv) Quality assurance methods 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used

5-vii) Access
"For this study, 2 separate interfaces for the participants were created on the SurveyMonkey platform (Momentive Inc)." 5-viii) Mode of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework "Intervention Group A In the web-based interface, intervention group A received a newly developed intervention created by the authors of this study with expertise in medical and health sciences education and research methodology. The educational intervention was written in the Croatian language. It consisted of 11 short educational texts on the methodology of producing SRs.
Intervention Group B Intervention group B was presented with the PRISMA checklist [22] for reporting on SRs (Multimedia Appendix 4) in their web-based interface, and the participants were asked to read it. It was presented to the participants in 11 separate sections to be as similar as possible to the number and form of the educational texts in intervention group A." 5-ix) Describe use parameters 5-x) Clarify the level of human involvement 5-xi) Report any prompts/reminders used "After the initial email was sent to the participants with the link to their respective study arm, 3 more reminders were sent to the participants 4 days apart." 5-xii) Describe any co-interventions (incl. training/support) "There were no conducted co-interventions." 6a) CONSORT: Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed "The primary outcome was the difference in the percentage of correct answers per participant in the postintervention questionnaire between intervention groups A and B.
Secondary Outcomes were the difference in the number of correct answers per participant in the pre-and postintervention questionnaires for the educational intervention group and the proportion of participants who correctly recognized SR abstracts" 6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed 6a-ii) Describe whether and how "use" (including intensity of use/dosage) was defined/measured/monitored 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained 6b) CONSORT: Any changes to trial outcomes after the trial commenced, with reasons "There were 8 eligible institutions in Croatia for this study, and we invited all of them." 7a) CONSORT: How sample size was determined 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size 7b) CONSORT: When applicable, explanation of any interim analyses and stopping guidelines "The primary outcome was the difference in the percentage of correct answers per participant in the postintervention questionnaire between intervention groups A and B.
Secondary Outcomes were the difference in the number of correct answers per participant in the pre-and postintervention questionnaires for the educational intervention group and the proportion of participants who correctly recognized SR abstracts" 8a) CONSORT: Method used to generate the random allocation sequence "The participants were randomized by simple randomization using the Randomizer website. After randomization, the participants were allocated to the study arms using a randomization sequence by a third person who was not included in other parts of the study." 8b) CONSORT: Type of randomisation; details of any restriction (such as blocking and block size) "The participants were randomized by simple randomization using the Randomizer website." 9) CONSORT: Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned "After randomization, the participants were allocated to the study arms using a randomization sequence by a third person who was not included in other parts of the study." 10) CONSORT: Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions "After randomization, the participants were allocated to the study arms using a randomization sequence by a third person who was not included in other parts of the study." 11a) CONSORT: Blinding -If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how 11a-i) Specify who was blinded, and who wasn't "The intervention was of such a nature that the participants could not be blinded.
MKM removed the participants'names and email addresses before the outcome assessor (IB) analyzed the data; thus, anonymized data were analyzed." 11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator"

11b) CONSORT: If relevant, description of the similarity of interventions
Not relevant in this manuscript. 12a) CONSORT: Statistical methods used to compare groups for primary and secondary outcomes "To determine the normality of the variables' distribution, we used the Kolmogorov-Smirnov test. Categorical data were presented as frequencies and percentages, and numerical values were presented as medians with IQR for variables not following normal distribution and as arithmetic means with IQR for variables following normal distribution. Differences between intervention groups A and B for categorical variables were tested using the chi-square test. To express the difference between groups, numerical values were tested with 2-tailed t tests for independent samples (for variables following normal distribution) and Mann-Whitney tests (for variables not following normal distribution). Pre-and postintervention differences were evaluated using the chisquare test for categorical variables and the t test for independent samples for numerical variables. The effect size for the primary outcome (the difference between the percentage of correct answers between groups in the postintervention questionnaire) was expressed using relative risk (RR) and 95% CI, as was the difference between the number of correct answers in the pre-and postintervention questionnaires in both groups. The effect size for the secondary outcome was expressed using odds ratio with 95% CI." 12a-i) Imputation techniques to deal with attrition / missing values Not applicable to our manuscript. 12b) CONSORT: Methods for additional analyses, such as subgroup analyses and adjusted analyses Not applicable to this manuscript. RESULTS 13a) CONSORT: For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome A total of 294 participants were assigned to the intervention A group and 295 participants were assigned to the intervention B group. 13b) CONSORT: For each group, losses and exclusions after randomisation, together with reasons A total of 81 participants form the group A and 88 participants from the group B were lost after randomization due to not signing into the platform. 13b-i) Attrition diagram 14a) CONSORT: Dates defining the periods of recruitment and follow-up "The link to participate in the study was sent via email to the addresses of the 589 students on June 7, 2021. The students were sent 3 reminders 4 days apart, and access to the web-based platforms was inactivated on June 20, 2021." 14a-i) Indicate if critical "secular events" fell into the study period 14b) CONSORT: Why the trial ended or was stopped (early) The trial did not end or was not stopped early.

15) CONSORT: A table showing baseline demographic and clinical characteristics for each group
The demographic characteristics are presented in Table 1 of the manuscript. 15-i) Report demographics associated with digital divide issues "The median length of working in health care was 9.9 years among participants who received the educational intervention and 9.8 years in the PRISMA checklist group. The median age of the participants in both groups was approximately 30 years, and >85% of the participants in both groups were women (educational intervention group: 140/162, 86.4%; PRISMA checklist group: 146/165, 88.5%;." 16a) CONSORT: For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups 16-i) Report multiple "denominators" and provide definitions "In the postintervention questionnaire, of the 1458 potential correct answers, there were 1086 (74.49%) correct answers to knowledge questions in the educational intervention group (162/327, 49.5%). In the PRISMA checklist group (165/327, 50.5%), of the 1485 potential correct answers, there were 900 (60.61%) correct answers (Table 2). Thus, the effect size for the difference in the number of correct answers to knowledge questions between groups was an RR of 1.23 (95% CI 1.17-1.29); that is, the educational intervention group had 23% (relative risk percentage) more correct answers in the postintervention questionnaire than the PRISMA checklist group." 16-ii) Primary analysis should be intent-to-treat 17a) CONSORT: For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval)