Cool but Counterproductive: Interactive, Web-Based Risk Communications Can Backfire
Background: Paper-based patient decision aids generally present risk information using numbers and/or static images. However, limited psychological research has suggested that when people interactively graph risk information, they process the statistics more actively, making the information more available for decision making. Such interactive tools could potentially be incorporated in a new generation of Web-based decision aids.
Objective: The objective of our study was to investigate whether interactive graphics detailing the risk of side effects of two treatments improve knowledge and decision making over standard risk graphics.
Methods: A total of 3371 members of a demographically diverse Internet panel viewed a hypothetical scenario about two hypothetical treatments for thyroid cancer. Each treatment had a chance of causing 1 of 2 side effects, but we randomly varied whether one treatment was better on both dimensions (strong dominance condition), slightly better on only one dimension (mild dominance condition), or better on one dimension but worse on the other (trade-off condition) than the other treatment. We also varied whether respondents passively viewed the risk information in static pictograph (icon array) images or actively manipulated the information by using interactive Flash-based animations of “fill-in-the-blank” pictographs. Our primary hypothesis was that active manipulation would increase respondents’ ability to recognize dominance (when available) and choose the better treatment.
Results: The interactive risk graphic conditions had significantly worse survey completion rates (1110/1695, 65.5% vs 1316/1659, 79.3%, P < .001) than the static image conditions. In addition, respondents using interactive graphs were less likely to recognize and select the dominant treatment option (234/380, 61.6% vs 343/465, 73.8%, P < .001 in the strong dominance condition).
Conclusions: Interactivity, however visually appealing, can both add to respondent burden and distract people from understanding relevant statistical information. Decision-aid developers need to be aware that interactive risk presentations may create worse outcomes than presentations of static risk graphic formats.
J Med Internet Res 2011;13(3):e60)
Ample evidence exists that even highly educated adults can have poor numeracy skills [- ]. As a result, patient decision aids and other patient communications that incorporate risk statistics will only be effective in improving patient decision making if they use design features that make these risk communications easier to understand. In this vein, researchers have evaluated the benefits of giving people more intuitive representations of risks by, for example, using frequencies instead of percentages [ - ] and testing a wide variety of visual displays, such as bar graphs, pie charts, and pictographs or icon arrays [ - ].
Improving risk-communication methods is also important because patients are likely to translate risk statistics into intuitive “gist” representations [, ] that may influence anxiety or worry, powerful emotions that have significant impacts on people’s responses to health risks and disease [ - ]. In fact, emotional responses may mediate cognitive risk perceptions or shape behavior independently, or both [ - ].
Recently, Natter and Berry suggested that communications that force the audience to actively process risk information may be more effective than more passive displays . In their study, participants were better calibrated in their perceptions of medication side effects when they created a bar graph of the risk instead of just viewing one. Similarly, Ancker and colleagues found that a Web-based, game-like, interactive risk graphic in which participants clicked in a matrix until they uncovered a risk event had the effect of reducing disparities in risk perceptions between high- and low-numeracy participants [ ]. While such interactive graphing tasks were formerly difficult to implement, advances in the interactive capabilities of the Internet provide an opportunity to integrate interactive graphing tasks into patient decision-support materials. Such exercises could be seen as one method of increasing patients’ active processing of risk information. Indeed, Ancker and colleagues also showed that their interactive risk graphics may elicit emotional responses that reflect the potential for increased understanding of actual risks and better ability to compare and contrast risks [ ].
We therefore hypothesized that a task designed for active processing might help patients not only to comprehend the statistics presented to them but also to integrate these facts into their decision making and thereby make better decisions.
To test the hypothesis that interactive graphing tasks could improve knowledge of risk information and decision making relevant to the risks, we conducted an Internet-administered experiment in which participants read a hypothetical treatment decision-making scenario that included information about the risks of 2 possible side effects. While some study participants passively viewed the risk information in state-of-the-art static risk images, others had to actively complete an interactive graphing task that asked them to fill in an icon array to match risk statistics that were provided numerically. We assessed whether the interactive task affected survey completion rates, treatment choices, and gist knowledge about the treatment options.
A stratified random sample of US adults age 21 years and older was selected from a panel of Internet users administered by Survey Sampling International (Shelton, CT, USA). To ensure at least moderate demographic diversity (but not representativeness) and offset large expected variations in response rates, we drew distinct subsamples by both age and race, and dynamically adjusted the number of email invitations in each demographic subsample until all quotas were achieved. Selected panel members received email invitations to complete the online survey. Upon completing the survey, participants were entered into both an instant-win contest and a monthly drawing administered by Survey Sampling International for modest prizes.
Design of the Study
Respondents read a short vignette in which they imagined being diagnosed with thyroid cancer and discussing treatment options with their doctor. The vignette discussed 2 types of radiation treatment, external beam therapy and seed therapy, which were described as being equally effective in treating the patient’s type of thyroid cancer. Both therapies were described as having a chance of causing 2 side effects: (1) fatigue, and (2) mouth and throat problems. We chose thyroid cancer as the disease context because its comparative unfamiliarity (versus, for example, breast or prostate cancer) meant that few study participants would be likely to have preconceived beliefs about treatment options or their associated risks.
Our primary research question was to determine whether an interactive graphing task would increase respondents’ ability to recognize a better treatment option (ie, one with lower risks of side effects) when such was available. To do so, we experimentally varied the content of the 2 survey pages that presented the risk of each side effect with the two treatment options. On those pages, we varied 2 factors in a 3 (risk levels) × 2 (graphic type) between-subjects design.
The risk-level manipulation is summarized in. By varying the likelihood of experiencing mouth or throat problems with beam therapy, we created 3 distinct decision contexts: (1) a trade-off condition, in which each treatment had a higher likelihood of one side effect and a lower likelihood of the other, (2) a mild dominance condition, in which seed therapy had a very slightly lower rate of fatigue and the same rate of mouth and throat problems, and (3) a strong dominance condition in which seed therapy had a lower rate of both side effects.
All risk information for the side effects was presented both numerically and in side-by-side graphics on sequential screens (first fatigue and then mouth and throat problems). Half of study participants viewed the risks displayed in static 100-unit icon arrays (also called pictographs:), a format demonstrated to improve risk communication in a variety of medical contexts [ , , - ]. The remaining participants received an interactive version of the same icon array format ( ), programmed in Flash (Adobe Systems Inc, San Jose, CA, USA). In the interactive exercise, the risk information was provided numerically but the graphics were initially blank (ie, all gray blocks). Participants were then instructed to use their mouse to click and/or drag in each graph to set it to the appropriate risk level. The graph continually adjusted to provide both visual and numerical feedback. (See for a movie demonstrating the interaction.) Participants were required to interact with and set the first of the 2 graphs on the page before they were allowed to go on in the study. The specific instructions given to participants are shown in . Note, however, that participants were not prevented from advancing if they did not set the second graph or were inaccurate in their graphing of the risks; minimal use of the interactive graphics was sufficient to allow them to continue in the study. This design received Institutional Review Board exempt status approval as anonymous survey research.
Measures and Covariates
Our primary outcome measure was the preferred treatment choice (beam or seed). We also asked respondents 2 gist knowledge questions in which they were to indicate which therapy had a higher risk of each of the side effects. In addition, we electronically tracked 2 measures of respondent burden: (1) time spent reading the static graphs pages or interacting with the interactive graphs pages, and (2) survey break-offs (to assess whether the interactive task annoyed people sufficiently to make them stop taking the survey).
Because individuals vary in their numeracy (ie, their facility and comfort with quantitative health information such as risk statistics), all study participants also completed the Subjective Numeracy Scale (SNS) . The SNS is a validated measure of quantitative ability and of preferences for receiving information in numerical form that has previously been shown to correlate with the ability to recall and comprehend both textual and graphical risk communications [ , , ]. A participant’s SNS score is calculated as his or her mean rating across the 8 SNS questions and ranges from 1 (least numerate) to 6 (most numerate). We also assessed participants’ level of education, which we model for analysis purposes as a 3-level variable: high school or less, some post-high school education but no bachelor’s degree, and bachelor’s degree or more, as well as standard demographics measures.
We used chi-square tests of proportions to test whether graph type affected treatment choices, knowledge recall, and survey discontinuation rates across the 3 risk-level conditions and logistic regression models that included both design factors, an interaction term, SNS score, and education level to assess the impact of these covariates on treatment choices. Because time spent on a given survey page has a highly skewed distribution, we then used Wilcoxon rank-sum tests to compare the distributions of time spent reading the static risk graphics versus completing the interactive graphing task. All analyses were performed using Stata (release 11; StataCorp LP, College Station, TX, USA), and all tests of significance were 2-sided and used alpha = .05.
In total, 3371 people age 21 years and older reached the survey website and viewed the first content page. Of these, 17 reported having been actually diagnosed with thyroid cancer and were excluded as having pre-existing knowledge of the relevant treatment options, leaving 3354 possible participants.
Overall, 2426 (72%) of participants completed the entire survey, including questions on demographics that came toward the end of the survey instrument. Characteristics for those participants who answered each demographic question are reported in. We observed a wide range of educational achievement, with 889 participants (36.7%) having a bachelor’s or higher college degree but also 441 (18.2%) having completed only high school or less education. The SNS numeracy measure showed high reliability (Cronbach alpha = .84). Mean SNS score was 4.63 (SD 1.0), with substantial variation (range 1.75–6.0). Because questions about participant demographics came at the end of the survey, we cannot know whether the demographics of those who dropped out differ from those who completed the survey.
Dropout Rates and Time Spent on Risk Graphic Pages
As shown in, participants randomly assigned to complete the interactive graphics task were significantly less likely to complete the survey than those randomly assigned to view static graphs (65.5% vs 79.3%). More detailed examination confirmed that this difference was specifically due to the interactive graphics task. Over 23% of participants in the interactive graphics condition dropped out of the survey during that section of the survey, while less than 4% of participants viewing static graphics dropped out on those pages. Even among those who did progress beyond that point in the survey, it took significantly longer to complete the interactive graphing task than to view the static graphs.
However problematic, the longer completion times and lower completion rates noted above for participants completing the interactive graphing task might be acceptable if it resulted in improved treatment choices among those who did complete the task. Unfortunately, such was not the case. As shown in, selection of the best (dominant) treatment option of seed therapy in the strong dominance condition was significantly higher among those in the static graphics group than in the interactive graphics group (343/465, 73.8% vs 234/380, 61.6%, χ21 = 14.3, P < .001, among only those participants who answered the choice question; 343/536, 64.0% vs 234/558, 41.9%, χ21 = 53.4, P < .001, in an intent-to-treat analysis that included all participants, including those who dropped out of the survey). There was no significant difference in treatment choices in the mild dominance condition, and a mild trend toward less selection of seed therapy by participants viewing static graphs versus interactive graphs in the trade-off condition.
In fact, across all 3 risk-level conditions, the interactive graphics condition made those participants less sensitive to variations in the risk of mouth and throat problems than were participants in the static graphs condition. A logistic regression analysis that included both experimental factors, as well as participant numeracy and education, confirmed a significant graph type × condition interaction (χ22 = 18.4, P < .001). In addition, participants with higher SNS scores were significantly more likely to choose the best treatment (odds ratio = 1.18 per step on the 1–6 scale, z = 3.78, P < .001), but participant education had no effect. More complicated models found no significant interaction between participants’ numeracy levels and the effect of graph type.
We also examined whether those participants in the interactive graphics condition who accurately graphed the risk statistics made different treatment choices from those who had difficulty completing this task. In the strong dominance condition (the only one that showed a significant difference overall between participants who viewed interactive and static graphics), study participants who accurately graphed all 4 side-effect risks (243, 64.0%, of the 380 who answered the treatment choice question) were significantly more likely to choose the dominant option of seed therapy (164/243, 67.5% vs 70/137, 51%, χ21 = 10.0, P = .002). We note, however, that optimal decision making remained at somewhat lower levels than observed in the static graphics condition (164/243, 67.5% vs 343/465, 73.8%, χ21 = 3.1, P = .08), thereby providing no evidence that even completely accurate use of an interactive graphing task would improve people’s treatment selections.
As shown in, study participants who had to complete interactive graphing tasks were also less able to identify which treatment had the higher risk of mouth or throat problems in the trade-off condition (43.8% vs 51.6%). Logistic regression analysis found this effect only marginally significant (odds ratio = 0.77, z = –1.77, P = .08) when controlling for the highly significant effect of numeracy (odds ratio = 1.52 per step on the 1–6 scale, z = 5.17, P < .001). There were no significant differences in gist knowledge about mouth and throat problems in the other 2 conditions. Nor were there any significant differences in gist knowledge about which treatment had a higher rate of fatigue (a side effect that did not vary across the 3 risk-level conditions.)
The promise of interactivity is to potentially encourage people not just to read relevant information but also to actively process it. Such active processing might help people to remember the information and use it to make better treatment decisions.
Unfortunately, the promise of interactive risk-communication exercises remained unfulfilled in our study. Our interactive task, which asked participants to graph pairs of risk statistics that were clearly presented in numerical format on the same page, increased survey dropouts and resulted in lower knowledge and poorer treatment choices than among those who viewed the information already displayed in equivalent static graphs.
Our study has several key limitations. First and foremost, we tested a single type of interactive task with a particular user interface, one that did not undergo external usability testing. It is certainly possible that other tasks or more intuitive interfaces could have improved the use and user experience of the interactive task and potentially thereby improved outcomes. Second, as noted above, we asked otherwise healthy adults to answer a hypothetical scenario in an Internet survey. As a result, it is likely that these participants were considerably less motivated to work out how to do the interactive task and to think about the risk information we provided than real patients would be when reading a detailed decision aid designed to inform them and help them make their own medical decisions. Yet, we observed no advantage of the interactive task even for those participants who completed it perfectly. It is hard to imagine that even high levels of motivation would be enough to not just offset the knowledge and decision-making deficits we observed but actually accrue significant benefits from completing the interactive graphing exercise.
Comparison with Prior Work
As Natter and Berry note, although other research exists that uses interactive graphics in risk communication , such research has not generally evaluated the interactive graphics against their passive counterparts. Accordingly, their study was the first to demonstrate the potential of interactive graphics over standard graphics by showing that people who draw a bar graph rather than simply viewed one had better recall of medication side effects [ ]. More recent research has suggested that interactive risk graphics in which participants explore a risk matrix until they find a risk event could also reduce numeracy disparities in risk perceptions [ ].
By contrast, our study showed no such positive effect. In fact, participants who were asked to interactively graph the risk information in our study did significantly worse on multiple outcome measures. This reversal may be partly accounted for by the fact that we used pictographs (icon arrays), which are less familiar to people than bar graphs, and also partly by the fact that our study was conducted online, whereas Natter and Berry’s used paper booklets. We also speculate post hoc that the novelty of the interactive task may have led participants to devote a majority of their attention and cognitive resources to figuring out what they were supposed to do. This effect may therefore have prevented our participants from engaging in the type of deeper, meaning-finding processing that we were hoping to stimulate. As noted above, a more in-depth, user-centered design process might have resulted in a better interface and hence reduced this problem.
Our different result may also derive from the fact that our task implemented a form of “teach-back” instructional methodology that encouraged participants to restate information they already had, whereas Ancker and colleagues’ research used an exploratory task that encouraged discovery of the risk itself. In qualitative analyses of focus group transcripts, Ancker and colleagues concluded that their interactive risk graphics elicited more emotional responses than static graphics did, with more participants expressing concern about large risks and/or relief about small ones . Their subsequent experimental study found no overall effect of graphic type on risk estimates or risk feelings [ ]. While our study did not ask participants about how the graphics made them feel about their risk, we did find that people were less likely to choose the clearly optimal option after having used an interactive risk graphic than they were with a standard risk graphic. This suggests that the increased emotional response observed by Ancker et al may be an artifact of their game-like task and that more straightforward types of interactivity may not lead to improved choices.
Research on other forms of interactivity in health education such as video games  and immersive 3-dimensional environments [ ] has suggested caution in considering how these applications might or might not improve outcomes in health education. Our results complement these prior findings by showing that interactive approaches to risk communication should come with similar caveats.
Interactive risk graphics are intriguing, but their use may be counterproductive to the purpose of effectively communicating to patients about health risks. While the movement of patient decision aids from booklet and DVD formats to online resources has created opportunities for greater technological sophistication, our research should serve as a cautionary tale to developers and educators seeking to ensure that their materials are “cutting edge.” Our teach-back interactive graphing task was intended to reinforce deeper cognitive processing of decision-relevant risk information, exactly the type of thinking most decision aids seek to promote in patients. Yet, in this case, our intervention backfired and created worse decision outcomes. More research is clearly needed to evaluate different types of interactive risk communications and to identify the design features of interactive exercises that lead to better results versus the features of those that do not. In the meantime, decision-aid designers should proceed with caution when considering the use of flashy risk graphics.
This research was supported by a grant from the Foundation for Informed Medical Decision Making (IIG-0126-1) to Dr. Zikmund-Fisher. Dr. Zikmund-Fisher is supported by a career development award from the American Cancer Society (MRSG-06-130-01-CPPB). The funding agreements ensured the authors’ independence in designing the study, interpreting the data, and publishing the report. The authors would like to thank Angela Fagerlin for helpful comments on an earlier draft of this manuscript, Bob Burbach and Mark Swanson for development of the Flash-based interactive graphics, and Nicole Exe for her exceptional research assistance.
Conflicts of Interest
Multimedia Appendix 1
MP4 movie of the interactive graphing task.MP4 File, 301KB
- Lipkus IM, Samsa G, Rimer BK. General performance on a numeracy scale among highly educated samples. Med Decis Making 2001;21(1):37-44. [Medline]
- Nelson W, Reyna VF, Fagerlin A, Lipkus I, Peters E. Clinical implications of numeracy: theory and practice. Ann Behav Med 2008 Jun;35(3):261-274. [CrossRef] [Medline]
- Peters E, Hibbard J, Slovic P, Dieckmann N. Numeracy skill and the communication, comprehension, and use of risk-benefit information. Health Aff (Millwood) 2007;26(3):741-748. [CrossRef] [Medline]
- Schwartz LM, Woloshin S, Black WC, Welch HG. The role of numeracy in understanding the benefit of screening mammography. Ann Intern Med 1997 Dec 1;127(11):966-972 [FREE Full text] [Medline]
- Waters EA, Weinstein ND, Colditz GA, Emmons K. Formats for improving risk communication in medical tradeoff decisions. J Health Commun 2006 Mar;11(2):167-182. [CrossRef] [Medline]
- Gigerenzer G. The psychology of good judgment: frequency formats and simple algorithms. Med Decis Making 1996;16(3):273-280. [Medline]
- Schapira MM, Nattinger AB, McHorney CA. Frequency or probability? A qualitative study of risk communication formats used in health care. Med Decis Making 2001;21(6):459-467. [Medline]
- Miron-Shatz T, Hanoch Y, Graef D, Sagi M. Presentation format affects comprehension and risk assessment: the case of prenatal screening. J Health Commun 2009;14(5):439-450. [CrossRef] [Medline]
- Waters EA, Weinstein ND, Colditz GA, Emmons KM. Reducing aversion to side effects in preventive medical treatment decisions. J Exp Psychol Appl 2007 Mar;13(1):11-21. [CrossRef] [Medline]
- Hawley ST, Zikmund-Fisher B, Ubel P, Jancovic A, Lucas T, Fagerlin A. The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns 2008 Dec;73(3):448-455. [CrossRef] [Medline]
- Lipkus IM, Hollands JG. The visual communication of risk. J Natl Cancer Inst Monogr 1999(25):149-163. [Medline]
- Stone ER, Yates JF, Parker AM. Effects of numerical and graphical displays on professed risk-taking behavior. J Exp Psychol Appl 1997;3(4):243-256. [CrossRef]
- Feldman-Stewart D, Brundage MD, Zotov V. Further insight into the perception of quantitative information: judgments of gist in treatment decisions. Med Decis Making 2007;27(1):34-43. [CrossRef] [Medline]
- Garcia-Retamero R, Galesic M. Who profits from visual aids: overcoming challenges in people's understanding of risks [corrected]. Soc Sci Med 2010 Apr;70(7):1019-1025. [CrossRef] [Medline]
- Reyna VF, Ellis SC. Fuzzy-trace theory and framing effects in children's risky decision making. Psychol Sci 1994;5(5):275-279. [CrossRef]
- Reyna VF, Hamilton AJ. The importance of memory in informed consent for surgical risk. Med Decis Making 2001;21(2):152-155. [Medline]
- McCaul KD, Tulloch HE. Cancer screening decisions. J Natl Cancer Inst Monogr 1999(25):52-58. [Medline]
- Rothman AJ, Kiviniemi MT. Treating people with information: an analysis and review of approaches to communicating health risk information. J Natl Cancer Inst Monogr 1999(25):44-51. [Medline]
- Cameron LD, Leventhal H, Love RR. Trait anxiety, symptom perceptions, and illness-related responses among women with breast cancer in remission during a tamoxifen clinical trial. Health Psychol 1998 Sep;17(5):459-469. [Medline]
- Lerman C, Trock B, Rimer BK, Boyce A, Jepson C, Engstrom PF. Psychological and behavioral implications of abnormal mammograms. Ann Intern Med 1991 Apr 15;114(8):657-661. [Medline]
- Lofters A, Juffs HG, Pond GR, Tannock IF. "PSA-itis": knowledge of serum prostate specific antigen and other causes of anxiety in men with metaststic prostate cancer. J Urol 2002 Dec;168(6):2516-2520. [CrossRef] [Medline]
- Trask PC, Paterson AG, Wang C, Hayasaka S, Milliron KJ, Blumberg LR, et al. Cancer-specific worry interference in women attending a breast and ovarian cancer risk evaluation program: impact on emotional distress and health functioning. Psychooncology 2001;10(5):349-360. [Medline]
- Slovic P, Peters E, Finucane ML, Macgregor DG. Affect, risk, and decision making. Health Psychol 2005 Jul;24(4 Suppl):S35-S40. [CrossRef] [Medline]
- Loewenstein GF, Weber EU, Hsee CK, Welch N. Risk as feelings. Psychol Bull 2001 Mar;127(2):267-286. [Medline]
- Finucane ML, Alhakami A, Slovic P, Johnson SM. The affect of heuristic judgments of risks and benefits. J Behav Decis Making 2000;13:1-17. [CrossRef]
- Natter HM, Berry DC. Effects of active information processing on the understanding of risk information. Appl Cogn Psychol 2005;19:123-135. [CrossRef]
- Ancker JS, Weber EU, Kukafka R. Effects of game-like interactive graphics on risk perceptions and decisions. Med Decis Making 2011;31(1):130-142. [CrossRef] [Medline]
- Ancker JS, Chan C, Kukafka R. Interactive graphics for expressing health risks: development and qualitative evaluation. J Health Commun 2009;14(5):461-475. [CrossRef] [Medline]
- Fagerlin A, Wang C, Ubel PA. Reducing the influence of anecdotal reasoning on people's health care decisions: is a picture worth a thousand statistics? Med Decis Making 2005;25(4):398-405. [CrossRef] [Medline]
- Zikmund-Fisher BJ, Ubel PA, Smith DM, Derry HA, McClure JB, Stark A, et al. Communicating side effect risks in a tamoxifen prophylaxis decision aid: the debiasing influence of pictographs. Patient Educ Couns 2008 Nov;73(2):209-214. [CrossRef] [Medline]
- Price M, Cameron R, Butow P. Communicating risk information: the influence of graphical display format on quantitative information perception-Accuracy, comprehension and preferences. Patient Educ Couns 2007 Dec;69(1-3):121-128. [CrossRef] [Medline]
- Zikmund-Fisher BJ, Fagerlin A, Ubel PA. Improving understanding of adjuvant therapy options by using simpler risk graphics. Cancer 2008 Dec 15;113(12):3382-3390 [FREE Full text] [CrossRef] [Medline]
- Galesic M, Garcia-Retamero R, Gigerenzer G. Using icon arrays to communicate medical risks: overcoming low numeracy. Health Psychol 2009 Mar;28(2):210-216. [CrossRef] [Medline]
- Fagerlin A, Zikmund-Fisher BJ, Ubel PA, Jankovic A, Derry HA, Smith DM. Measuring numeracy without a math test: development of the Subjective Numeracy Scale. Med Decis Making 2007;27(5):672-680. [CrossRef] [Medline]
- Zikmund-Fisher BJ, Smith DM, Ubel PA, Fagerlin A. Validation of the Subjective Numeracy Scale: effects of low numeracy on comprehension of risk communications and utility elicitations. Med Decis Making 2007;27(5):663-671. [CrossRef] [Medline]
- Zikmund-Fisher BJ, Fagerlin A, Ubel PA. A demonstration of ''less can be more'' in risk graphics. Med Decis Making 2010;30(6):661-671. [CrossRef] [Medline]
- Woloshin S, Schwartz LM, Byram S, Fischhoff B, Welch HG. A new scale for assessing perceptions of chance: a validation study. Med Decis Making 2000;20(3):298-307. [Medline]
- Annetta LA, Minogue J, Holmes SY. Cheng M-T. Investigating the impact of video games on high school students' engagement and learning about genetics. Computers & Education 2009;53(1):74-85. [CrossRef]
- Hansen MM. Versatile, immersive, creative and dynamic virtual 3-D healthcare learning environments: a review of the literature. J Med Internet Res 2008;10(3):e26 [FREE Full text] [CrossRef] [Medline]
|SNS: Subjective Numeracy Scale|
Edited by G Eysenbach; submitted 05.10.10; peer-reviewed by J Ancker, T Miron-Shatz; comments to author 26.01.11; revised version received 24.02.11; accepted 05.05.11; published 25.08.11
©Brian J Zikmund-Fisher, Mark Dickson, Holly O Witteman. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 25.08.2011.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.