How the “Understanding Research Evidence” Web-Based Video Series From the National Collaborating Centre for Methods and Tools Contributes to Public Health Capacity to Practice Evidence-Informed Decision Making: Mixed-Methods Evaluation

Background The National Collaborating Centre for Methods and Tools (NCCMT) offers workshops and webinars to build public health capacity for evidence-informed decision-making. Despite positive feedback for NCCMT workshops and resources, NCCMT users found key terms used in research papers difficult to understand. The Understanding Research Evidence (URE) videos use plain language, cartoon visuals, and public health examples to explain complex research concepts. The videos are posted on the NCCMT website and YouTube channel. Objective The first four videos in the URE web-based video series, which explained odds ratios (ORs), confidence intervals (CIs), clinical significance, and forest plots, were evaluated. The evaluation examined how the videos affected public health professionals’ practice. A mixed-methods approach was used to examine the delivery mode and the content of the videos. Specifically, the evaluation explored (1) whether the videos were effective at increasing knowledge on the four video topics, (2) whether public health professionals were satisfied with the videos, and (3) how public health professionals applied the knowledge gained from the videos in their work. Methods A three-part evaluation was conducted to determine the effectiveness of the first four URE videos. The evaluation included a Web-based survey, telephone interviews, and pretest and posttests, which evaluated public health professionals’ experience with the videos and how the videos affected their public health work. Participants were invited to participate in this evaluation through various open access, public health email lists, through informational flyers and posters at the Canadian Public Health Association (CPHA) conference, and through targeted recruitment to NCCMT’s network. Results In the Web-based surveys (n=46), participants achieved higher scores on the knowledge assessment questions from watching the OR (P=.04), CI (P=.04), and clinical significance (P=.05) videos but not the forest plot (P=.12) video, as compared with participants who had not watched the videos. The pretest and posttest (n=124) demonstrated that participants had a better understanding of forest plots (P<.001) and CIs (P<.001) after watching the videos. Due to small sample size numbers, there were insufficient pretest and posttest data to conduct meaningful analyses on the clinical significance and OR videos. Telephone interview participants (n=18) thought the videos’ use of animation, narration, and plain language was appropriate for people with different levels of understanding and learning styles. Participants felt that by increasing their understanding of research evidence, they could develop better interventions and design evaluations to measure the impact of public health initiatives. Conclusions Overall, the results of the evaluation showed that watching the videos resulted in an increase in knowledge, and participants had an overall positive experience with the URE videos. With increased competence in using the best available evidence, professionals are empowered to contribute to decisions that can improve health outcomes of communities.


What is clinical significance?
a. Whether the effect found in the study is big enough to justify investing in the intervention b. When an intervention only works in a clinical setting c. Confidence that the results are real d. The confidence interval in a study 2. What is the most relevant information you need to calculate clinical significance? Select all that apply.
a. Professional judgment b. Effect size c. Confidence intervals d. Both a and c

Odds Ratios
1. What does an odds ratio tell you? a. Whether or not an intervention works b. The sample population size compared to the general population c. Odds that an action/intervention will lead to a particular outcome d. Both a and c 2. What does an odds ratio greater than 1 mean? a. The outcome of interest is more likely in the intervention group. b. The outcome of interest is less likely in the intervention group

Confidence Intervals
1. What is a confidence interval? a. Indicates range of intervention effectiveness b. Indicates consistency of the results of the study to the results for a hypothetical sample c. Indicates we can be 95% certain of results of an individual study d. Both a and b 2. What is the interpretation of the result when a confidence interval includes the number of no effect? a. The study results of an intervention may be due to chance and may not be real b. The range of intervention effectiveness is broadened c. The effect size in the study will be large enough to justify investing in the intervention d. It will be more difficult to decide whether or not to implement an intervention 3. How does a confidence interval help you determine whether or not to apply an intervention? a. Helps determine how confident you can be in the results b. Describes the effects an intervention has across various samples of the population, not just the effect for one sample c. Helps determine statistical and clinical significance d. All of the above

Pretest and Posttest Knowledge Assessment Questions
The following questions were included in the pretest and posttest. The questions assessed participants' knowledge of the concepts covered in the URE videos on odds ratio, clinical significance, forest plots, and confidence intervals.

Odds Ratios
1. The odds ratio tells you whether or not an intervention works, and the effect size of the intervention. a. True b. False 2. What does an odds ratio greater than 1 mean? c. The outcome of interest is more likely in the intervention group. d. The outcome of interest is less likely in the intervention group.
3. What does an odds ratio of 1 mean? a. The intervention had no effect b. The outcome of interest is less likely in the intervention group 4. In this hypothetical study, the effectiveness of a citywide health promotion text-message intervention to increase helmet-use in daily bike commuters is being tested.
What is the odds ratio in this scenario?
Intervention ( 5. In a hypothetical study on the effect of a health promotion intervention to encourage lowrisk drinking practices for women, the odds ratio is 0.87. Which of the following statements correctly interprets this finding?
a. Women who received the health promotion intervention were more likely to practice low-risk drinking b. Women who received the health promotion intervention were less likely to practice low-risk drinking c. Women who did not receive the health promotion intervention were less likely to practice low-risk drinking d. The health promotion intervention had no effect on women's low-risk drinking practices

What is clinical significance?
a. Whether the effect found in the study is big enough to justify investing in the intervention b. When an intervention only works in a clinical setting c. The effect found in the study is real d. The confidence interval in a study 2. What is the most relevant information you need to calculate clinical significance? Select all that apply. a. Professional judgment b. Effect size c. Odds ratios/relative risk d. Confidence intervals 3. What factors determine clinical significance? Select all that apply. a. The resources available to the practitioner b. The range of intervention effect sizes stated by the confidence interval in the research evidence c. The intervention effect size determined by the odds ratios/relative risks in the research evidence d. The characteristics of the sample population in the research evidence 4. In a hypothetical example, you find a systematic review that found that health promotion textmessages were very effective in increasing bike helmet use amongst bike commuters in the city. The study showed that there was a 50-80% increase in bike helmet use after exposure to the text-message intervention.
When further exploring this intervention, you find text-messages are not included in many cellphone plans in your city. The health promotion text messages would cost bike commuters in your city $0.75 for each text message they receive.
In this situation, are health promotion text-messages a clinically significant intervention? a. Yes b. No c. Not enough information provided to determine the answer 5. In another hypothetical example, you find a meta-analysis that found that a snow shoveling and salting program reduces hip injuries in seniors by 90%.
You have decided that you can design a program that pairs volunteers with seniors to help them shovel and salt their walkways and sidewalks when it snows.
In this situation, is the snow-shoveling and salting a clinically significant intervention? e. Yes f. No g. Not enough information provided to determine the answer

Confidence Intervals
1. The confidence interval shows the range of effects an intervention can have, if the same study was replicated 100 times. a. True b. False 2. Is it necessary to consider both p-values and confidence intervals? a. Yes b. No 3. How can you tell if the finding in a study is statistically significant by looking at the 95% confidence interval?
a. The finding is statistically significant if the 95% confidence interval includes the relative risk or odds ratio in its range. b. The finding is statistically significant if the 95% confidence interval includes the number of no effect (1) in its range, and the relative risk/odds ratio is above 1 c. The finding is statistically significant if the 95% confidence interval does not include the number of no effect (1) in its range. d. The finding is statistically significant if the 95% confidence interval does not include the number of no effect (1) in its range, and the relative risk/odds ratio is above 1.
4. What can the confidence interval tell you about an intervention? a. The effect the intervention will have on a specific population b. The different outcomes from an intervention c. The statistical significance of an intervention d. The clinical significance of an intervention 5. In a hypothetical example of a text-message intervention to increase helmet use amongst daily bike commuters, if the confidence interval indicates an effect between a 10% reduction in bike helmet use and a 60% increase in bike helmet use. Would you implement the text-message intervention? a. Yes b. No

Forest Plots
1. What does a forest plot tell you? Select all that apply e. The range of intervention effects from the individual studies f. Whether the intervention is statistically significant g. The overall effect size of the intervention h. All of the above 2. How can you determine the range of effects an intervention can have from a forest plot? e. The vertical points of the diamond f. The outcome measures on the horizontal axis g. The left and right points of the diamond h. The individual points in the forest plot