This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
The GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach is a system for transparent evaluation of the certainty of evidence used in clinical practice guidelines and systematic reviews. GRADE is a key part of evidence-based medicine (EBM) training of health care professionals.
This study aimed to compare web-based and face-to-face methods of teaching the GRADE approach for evidence assessment.
A randomized controlled trial was conducted on 2 delivery modes of GRADE education integrated into a course on research methodology and EBM with third-year medical students. Education was based on the Cochrane Interactive Learning “Interpreting the findings” module, which had a duration of 90 minutes. The web-based group received the web-based asynchronous training, whereas the face-to-face group had an in-person seminar with a lecturer. The main outcome measure was the score on a 5-question test that assessed confidence interval interpretation and overall certainty of evidence, among others. Secondary outcomes included writing a recommendation for practice and course satisfaction.
In all, 50 participants received the web-based intervention, and 47 participants received the face-to-face intervention. The groups did not differ in the overall scores for the Cochrane Interactive Learning test, with a median of 2 (95% CI 1.0-2.0) correct answers for the web-based group and 2 (95% CI 1.3-3.0) correct answers for the face-to-face group. Both groups gave the most correct answers to the question about rating a body of evidence (35/50, 70% and 24/47, 51% for the web-based and face-to-face group, respectively). The face-to-face group better answered the question about the overall certainty of evidence question. The understanding of the Summary of Findings table did not differ significantly between the groups, with a median of 3 correct answers to 4 questions for both groups (
Training in the GRADE approach could be equally effective when delivered asynchronously on the web or face-to-face.
Open Science Framework akpq7; https://osf.io/akpq7/
The extent of confidence in the desirable effects of an intervention outweighing the undesirable ones is a valuable indicator in the strength of recommendations for clinical practice [
GRADE is not the first nor the only approach for assessing the certainty of evidence and assigning the strength of recommendations. It is, however, the approach used in Cochrane systematic reviews and many clinical practice guidelines. The GRADE approach includes a Summary of Findings (SoF) table to make the process of judging evidence and translating it into a recommendation more accessible for a broader audience, primarily for end users—clinicians, patients, and policy makers. It is a systematic, transparent, and concise report of key information that includes the certainty of evidence and the effect size of an intervention used for each outcome and across outcomes [
To effectively apply clinical practice guidelines and other summarized formats of evidence, health care providers need to have the evidence-based medicine (EBM) skills necessary for understanding and the application of clinical practice guidelines. Aside from occasional specialized courses and a short video series [
Web-based education uses web-based technologies for knowledge and skills improvement. It can be asynchronous, in which users can individually access it anytime and progress through it at their own pace. It can also be synchronous, in which users have to access it at certain times, usually in some form of webinars. Web-based educational interventions have shown noninferior results in learning and participant satisfaction outcomes compared to face-to-face learning in medicine, including communication skills and cardiology [
The aim of this study was to determine the effectiveness of a web-based educational intervention for the GRADE approach to evidence assessment, compared to traditional classroom education, in terms of knowledge and the understanding of the SoF table.
This was a parallel-group randomized controlled trial. Participants were third-year medical students in Croatian- and English-language programs at the University of Split School of Medicine. Students were attending a mandatory course on research methodology and EBM, described in a previously published study [
The web-based educational intervention was based on the Cochrane Interactive Learning (CIL) module 7, titled “Interpreting the findings” [
After 90 minutes, the participants took the same test, hosted on the SurveyMonkey platform (SurveyMonkey Inc) [
The first part of the SurveyMonkey test was a brief sociodemographic questionnaire, which included questions on participants age, gender, level of education, current research activities, and authorship of research publications. The participants were also asked to assess their knowledge of the GRADE approach (ranging from 1=little to none to 5=excellent), as well as their familiarity with Cochrane and systematic reviews (from 1=not at all to 5=extremely familiar).
After that, the participants took a test that assessed their knowledge of the GRADE approach. Five multiple-choice questions on statistical terms and their evaluation were taken from the official assessment for the CIL module [
The final part of the test evaluated the participants’ understanding of an SoF table, which was evaluated with 4 open-ended questions linked to an SoF example [
The SoF table and the questions from the test are available in
The primary outcome for this study was the knowledge measured by the 5 questions from the formal CIL module. The knowledge was measured in 2 ways: as the overall scores for the test and the number of students correctly answering each question.
There were 3 secondary outcomes:
The understanding of the SoF table was measured by 4 questions related to an SoF table example. The results were expressed as the number and percentage of students with correct answer to individual questions and the total score for the whole group.
Participants’ satisfaction with and opinion about the course was measured using 10 questions with Likert-type statements, with scoring ranging from 1=I do not agree at all to 7=I fully agree.
The style of writing of a recommendation for clinical practice in the answer to the first of the 4 questions about the SoF table: we assessed the style according to the National Institute for Health and Care Excellence (NICE) instructions for writing recommendations [
The
The opinion of the students about the
Two independent assessors (RT and TPP) rated all of the responses. Inconsistencies in their ratings were resolved with the help of a third author. κ statistics were used to determine the level of agreement for each of the 3 categories.
Based on the primary outcome and the assumption that there would be no significant differences between groups, we calculated the minimal sample size using a web-based calculator [
We used a simple randomization method [
Sociodemographic characteristics of participants are presented as absolute numbers and percentages. Group results are presented as medians and 95% Cis. The distribution of results was tested using the Kolmogorov-Smirnov test, and group results were compared using the Mann-Whitney
The study was approved by the Ethics Committee of the University of Split School of Medicine (class 003-08/19-03/0003; registration 2181-198-03-04-19-0044). The participants gave informed consent, and the data were kept according to the General Data Protection Regulation.
The participant flow diagram is shown in
Flow diagram of the participants in the study.
Demographic data, previous research experience, and self-assessed knowledge of the GRADEa approachb.
Item | Web-based training (n=50) | Face-to-face training (n=47) |
Gender (female; total: n=90, web-based training: n=48, face-to-face training: n=42), n (%) | 29 (60) | 27 (64) |
Age (years; total: n=89, web-based training: n=47, face-to-face training: n=42), median (IQR) | 21.0 (21-23) | 21.0 (21-23) |
Level of completed education (high school; total: n=89, web-based training: n=47, face-to-face training: n=42), n (%) | 45 (96) | 40 (95) |
Are you currently involved in research activities? (yes; total: n=85, web-based training: n=48, face-to-face training: n=37), n (%) | 1 (2) | 0 (0) |
Authorship of a research publication in the last 5 years (yes; total: n=80, web-based training: n=47, face-to-face training: n=33), n (%) | 1 (2) | 1 (3) |
Authorship of a systematic review (yes; total: n=88, web-based training: n=47, face-to-face training: n=41), n (%) | 1 (2) | 0 (0) |
Authorship of a clinical practice guideline (yes; total: n=89, web-based training: n=48, face-to-face training: n=41), n (%) | 0 (0) | 0 (0) |
How familiar are you with Cochrane collaboration? (1=not at all, 5=extremely familiar; n=89), median (95% CI) | 2.0 (2-2) | 2.0 (2-2) |
How would you grade your knowledge of GRADE approach? (1=very low, 5=very high; n=88), median (95% CI) | 2.0 (2-2) | 2.0 (2-2) |
aGRADE: Grading of Recommendations Assessment, Development and Evaluation.
bThe numbers in parentheses indicate the number of responses in the questionnaire.
Participants’ satisfaction with the training session, presented as median scores with 95% CIa.
Item | Web-based training (n=50), median (95% CI) | Face-to-face training (n=47), median (95% CI) |
Overall, I am satisfied with the course (n=85) | 5 (5-5) | 5 (5-6) |
This course was really useful (n=87) | 5 (4-5) | 5 (5-5.8) |
This is a good way for learning GRADEb approach for quality of evidence (n=87) | 5 (4-5) | 5 (4-5) |
This course helped me to better understand the concepts related to GRADE (n=87) | 5 (4-5) | 4 (4-5) |
The course covered too much content in a short period of time (n=88) | 4 (4-5) | 4 (3.2-5) |
I think there was sufficient amount of interaction during this course (n=86) | 5 (5-6) | 5 (4-6) |
I would recommend this course to my colleagues (n=85) | 5 (4-5.1) | 5 (4-5.1) |
I did not find this course useful (n=84) | 2 (2-4) | 2 (2-3) |
In future, I will apply what I learned at this course in my work and research (n=85) | 5 (5-5) | 5 (5-5) |
In future, I will learn more about interpreting and grading the quality of evidence (n=84) | 5 (4-6) | 5 (4.8-5.2) |
aThe numbers in parentheses indicate the number of responses in the questionnaire.
bGRADE: Grading of Recommendations Assessment, Development and Evaluation.
The groups did not differ in the overall scores for the CIL test (
Understanding of the SoF table also did not differ significantly between the groups (median of 3 correct questions to 4 questions for both groups;
For the analysis of how students phrased their recommendation for practice (
Number (%) of correct answers and overall score for the 5 questions of the Cochrane Interactive Learning test.
Item | Web-based training (n=50) | Face-to-face training (n=47) | |
Understanding of confidence intervals in the interpretation of results of meta-analysis, n (%) | 18 (36) | 16 (34) | >.99 |
Identify ways of re-expressing the standardized mean difference, n (%) | 14 (28) | 25 (53) | .014 |
Interpret a funnel plot asymmetry, n (%) | 9 (18) | 12 (26) | .461 |
Determine the overall certainty of the evidence, n (%) | 15 (30) | 27 (57) | .008 |
Decide on rating up a body of evidence, n (%) | 35 (70) | 24 (51) | .064 |
Overall score, median (95% CI) | 2 (1.0-2.0) | 2 (1.3-3.0) | .251b |
aFischer exact test.
bMann-Whitney
Number (%) of correct answers and overall score on 4 test questions related to the Summary of Findings table.
Item | Web-based training (n=50) | Face-to-face training (n=47) | |
Based on this information, how would you formulate a recommendation for clinical practice? (would recommend; total: n=83, web-based training: n=48, face-to-face training: n=35), n (%) | 27 (56) | 15 (43) | .270 |
Would you consider any subgroups of patients, and if so, how?, n (%) | 24 (48) | 22 (47) | >.99 |
How many participants were there in trials that assessed death as an outcome?, n (%) | 39 (78) | 30 (64) | .178 |
Why was the quality of evidence for hospitalized children graded as very low?, n (%) | 31 (62) | 39 (83) | .025 |
Overall score, median (95% CI) | 3 (2-3) | 2 (2-2) | .35b |
aFischer exact test.
bMann-Whitney
Number (%) of students who used specific writing style in their clinical practice recommendation based on the Summary of Findings table in the testa.
Category of writing recommendation and element | Web-based training (n=50), n (%) | Face-to-face training (n=47), n (%) | ||
|
||||
|
Active verb | 4 (8) | 0 (0) | .118 |
|
Target population | 24 (48) | 10 (21) | .010 |
|
Setting (time or context) | 2 (4) | 6 (13) | .151 |
|
28 (56) | 32 (68) | .296 | |
|
||||
|
Offer | 31 (62) | 27 (57) | .683 |
|
Give and prescribe | 10 (20) | 1 (2) | .008 |
|
No recommendation | 9 (18) | 19 (40) | .024 |
aStudents’ recommendations written in the answer to the first Summary of Findings question were assessed according to the presence of categories from the National Institute for Health and Care Excellence (NICE) writing recommendations. The number of categories was greater than the number of students as their recommendation could include more than one category element of the writing style.
bFischer exact test.
Our study showed that web-based education about GRADE methodology may not be different to face-to-face education, as measured by the CIL module overall test results. The face-to-face group was better at assessing evidence using the risk of bias. There were no differences between the groups in the overall understanding of the SoF table. Both groups had high levels of satisfaction with the intervention. These results should be evaluated in the context of additional educational resources in EBM courses for medical school students and taking into account a limited sample size.
The satisfaction with the course was high in both groups, and participants found both educational interventions to be sufficiently interactive. It has been shown that participants’ satisfaction influences their academic scores [
There were no differences between the groups for the overall test results and SoF table understanding. Although median scores were low for both, they were lower for the 5 questions on methodology. These questions might be too advanced for the third-year medical students. Some of the individual questions had better response rates. The participants in the face-to-face group better answered the question on assessing the overall certainty of the evidence, taking into account the risk of bias, as well as other domains. Previous research in risk-of-bias education involved doctoral students [
Both groups had high scores in the understanding of the SoF table, which is consistent with previous research on understanding evidence presented in this way [
Clear and understandable communication of evidence that is important for clinical practice is a previously recognized priority [
Students often used scientific language and formed their answers as conclusions without elements of a clinical decision. Such distancing from recommending a clear action or against one might be explained as the result of students’ lack of clinical experience but also as a part of the culture of defensive medicine, in which medical professionals avoid a decisive action due to the fear of complications and responsibility [
This study included a sample of third-year medical students, with limited clinical experience. Clinical experience might alter the perception of outcome importance and the severity of unwanted effects, both of which can influence recommendations for clinical practice. This trial did not include clinically experienced medical students or other health care workers, and its results might not translate to such populations. Another possible limitation of this study is that it did not involve an official GRADE training. There are no official criteria or consensus for defining what constitutes a GRADE methodologist—someone who support the creation of a guideline or help systematic review authors in evidence assessment [
EBM skills are necessary for decision-making in health care, but the transfer of this knowledge to practice is often inadequate [
Summary of Findings table and questions from the test.
CONSORT checklist.
Cochrane Interactive Learning
evidence-based medicine
Grading of Recommendations Assessment, Development and Evaluation
National Institute for Health and Care Excellence
Summary of Findings
This study was funded by the Croatian Science Foundation (Professionalism in Health - Decision making in practice and research [ProDeM]; grant IP-2019-04-4882). The funder had no role in the design of this study during its execution and data interpretation.
The data sets generated during this study are available from the first author (RT) on reasonable request.
AM and TPP gave the idea for the initial design of the study. RT and TPP developed the protocol. RT and TPP conducted the trial and collected the data. RT conducted statistical analysis, and all authors interpreted the results. RT drafted the manuscript. All authors revised the manuscript critically for important intellectual content and approved the final version of the manuscript for submission.
All authors are members of Cochrane.
This randomized study was retrospectively registered. The editor granted an exception from ICMJE rules mandating prospective registration of randomized trials since the trial was not health-related. However, readers are advised to carefully assess the validity of any potential explicit or implicit claims related to primary outcomes or effectiveness, as retrospective registration does not prevent authors from changing their outcome measures retrospectively.