A stratified random sample of US adults age 21 years and older was selected from a panel of Internet users administered by Survey Sampling International (Shelton, CT, USA). To ensure at least moderate demographic diversity (but not representativeness) and offset large expected variations in response rates, we drew distinct subsamples by both age and race, and dynamically adjusted the number of email invitations in each demographic subsample until all quotas were achieved. Selected panel members received email invitations to complete the online survey. Upon completing the survey, participants were entered into both an instant-win contest and a monthly drawing administered by Survey Sampling International for modest prizes.

Respondents read a short vignette in which they imagined being diagnosed with thyroid cancer and discussing treatment options with their doctor. The vignette discussed 2 types of radiation treatment, external beam therapy and seed therapy, which were described as being equally effective in treating the patient’s type of thyroid cancer. Both therapies were described as having a chance of causing 2 side effects: (1) fatigue, and (2) mouth and throat problems. We chose thyroid cancer as the disease context because its comparative unfamiliarity (versus, for example, breast or prostate cancer) meant that few study participants would be likely to have preconceived beliefs about treatment options or their associated risks.

Our primary research question was to determine whether an interactive graphing task would increase respondents’ ability to recognize a better treatment option (ie, one with lower risks of side effects) when such was available. To do so, we experimentally varied the content of the 2 survey pages that presented the risk of each side effect with the two treatment options. On those pages, we varied 2 factors in a 3 (risk levels) × 2 (graphic type) between-subjects design.

The risk-level manipulation is summarized in Table 1. By varying the likelihood of experiencing mouth or throat problems with beam therapy, we created 3 distinct decision contexts: (1) a trade-off condition, in which each treatment had a higher likelihood of one side effect and a lower likelihood of the other, (2) a mild dominance condition, in which seed therapy had a very slightly lower rate of fatigue and the same rate of mouth and throat problems, and (3) a strong dominance condition in which seed therapy had a lower rate of both side effects.

All risk information for the side effects was presented both numerically and in side-by-side graphics on sequential screens (first fatigue and then mouth and throat problems). Half of study participants viewed the risks displayed in static 100-unit icon arrays (also called pictographs: Figure 1), a format demonstrated to improve risk communication in a variety of medical contexts [9,13,29-33]. The remaining participants received an interactive version of the same icon array format (Figure 2), programmed in Flash (Adobe Systems Inc, San Jose, CA, USA). In the interactive exercise, the risk information was provided numerically but the graphics were initially blank (ie, all gray blocks). Participants were then instructed to use their mouse to click and/or drag in each graph to set it to the appropriate risk level. The graph continually adjusted to provide both visual and numerical feedback. (See Multimedia Appendix 1 for a movie demonstrating the interaction.) Participants were required to interact with and set the first of the 2 graphs on the page before they were allowed to go on in the study. The specific instructions given to participants are shown in Figure 1. Note, however, that participants were not prevented from advancing if they did not set the second graph or were inaccurate in their graphing of the risks; minimal use of the interactive graphics was sufficient to allow them to continue in the study. This design received Institutional Review Board exempt status approval as anonymous survey research.

Our primary outcome measure was the preferred treatment choice (beam or seed). We also asked respondents 2 gist knowledge questions in which they were to indicate which therapy had a higher risk of each of the side effects. In addition, we electronically tracked 2 measures of respondent burden: (1) time spent reading the static graphs pages or interacting with the interactive graphs pages, and (2) survey break-offs (to assess whether the interactive task annoyed people sufficiently to make them stop taking the survey).

Because individuals vary in their numeracy (ie, their facility and comfort with quantitative health information such as risk statistics), all study participants also completed the Subjective Numeracy Scale (SNS) [34]. The SNS is a validated measure of quantitative ability and of preferences for receiving information in numerical form that has previously been shown to correlate with the ability to recall and comprehend both textual and graphical risk communications [32,35,36]. A participant’s SNS score is calculated as his or her mean rating across the 8 SNS questions and ranges from 1 (least numerate) to 6 (most numerate). We also assessed participants’ level of education, which we model for analysis purposes as a 3-level variable: high school or less, some post-high school education but no bachelor’s degree, and bachelor’s degree or more, as well as standard demographics measures.

We used chi-square tests of proportions to test whether graph type affected treatment choices, knowledge recall, and survey discontinuation rates across the 3 risk-level conditions and logistic regression models that included both design factors, an interaction term, SNS score, and education level to assess the impact of these covariates on treatment choices. Because time spent on a given survey page has a highly skewed distribution, we then used Wilcoxon rank-sum tests to compare the distributions of time spent reading the static risk graphics versus completing the interactive graphing task. All analyses were performed using Stata (release 11; StataCorp LP, College Station, TX, USA), and all tests of significance were 2-sided and used alpha = .05.

In total, 3371 people age 21 years and older reached the survey website and viewed the first content page. Of these, 17 reported having been actually diagnosed with thyroid cancer and were excluded as having pre-existing knowledge of the relevant treatment options, leaving 3354 possible participants.

Overall, 2426 (72%) of participants completed the entire survey, including questions on demographics that came toward the end of the survey instrument. Characteristics for those participants who answered each demographic question are reported in Table 2. We observed a wide range of educational achievement, with 889 participants (36.7%) having a bachelor’s or higher college degree but also 441 (18.2%) having completed only high school or less education. The SNS numeracy measure showed high reliability (Cronbach alpha = .84). Mean SNS score was 4.63 (SD 1.0), with substantial variation (range 1.75–6.0). Because questions about participant demographics came at the end of the survey, we cannot know whether the demographics of those who dropped out differ from those who completed the survey.

The promise of interactivity is to potentially encourage people not just to read relevant information but also to actively process it. Such active processing might help people to remember the information and use it to make better treatment decisions.

Unfortunately, the promise of interactive risk-communication exercises remained unfulfilled in our study. Our interactive task, which asked participants to graph pairs of risk statistics that were clearly presented in numerical format on the same page, increased survey dropouts and resulted in lower knowledge and poorer treatment choices than among those who viewed the information already displayed in equivalent static graphs.

Our study has several key limitations. First and foremost, we tested a single type of interactive task with a particular user interface, one that did not undergo external usability testing. It is certainly possible that other tasks or more intuitive interfaces could have improved the use and user experience of the interactive task and potentially thereby improved outcomes. Second, as noted above, we asked otherwise healthy adults to answer a hypothetical scenario in an Internet survey. As a result, it is likely that these participants were considerably less motivated to work out how to do the interactive task and to think about the risk information we provided than real patients would be when reading a detailed decision aid designed to inform them and help them make their own medical decisions. Yet, we observed no advantage of the interactive task even for those participants who completed it perfectly. It is hard to imagine that even high levels of motivation would be enough to not just offset the knowledge and decision-making deficits we observed but actually accrue significant benefits from completing the interactive graphing exercise.

As Natter and Berry note, although other research exists that uses interactive graphics in risk communication [37], such research has not generally evaluated the interactive graphics against their passive counterparts. Accordingly, their study was the first to demonstrate the potential of interactive graphics over standard graphics by showing that people who draw a bar graph rather than simply viewed one had better recall of medication side effects [26]. More recent research has suggested that interactive risk graphics in which participants explore a risk matrix until they find a risk event could also reduce numeracy disparities in risk perceptions [27].

By contrast, our study showed no such positive effect. In fact, participants who were asked to interactively graph the risk information in our study did significantly worse on multiple outcome measures. This reversal may be partly accounted for by the fact that we used pictographs (icon arrays), which are less familiar to people than bar graphs, and also partly by the fact that our study was conducted online, whereas Natter and Berry’s used paper booklets. We also speculate post hoc that the novelty of the interactive task may have led participants to devote a majority of their attention and cognitive resources to figuring out what they were supposed to do. This effect may therefore have prevented our participants from engaging in the type of deeper, meaning-finding processing that we were hoping to stimulate. As noted above, a more in-depth, user-centered design process might have resulted in a better interface and hence reduced this problem.

Our different result may also derive from the fact that our task implemented a form of “teach-back” instructional methodology that encouraged participants to restate information they already had, whereas Ancker and colleagues’ research used an exploratory task that encouraged discovery of the risk itself. In qualitative analyses of focus group transcripts, Ancker and colleagues concluded that their interactive risk graphics elicited more emotional responses than static graphics did, with more participants expressing concern about large risks and/or relief about small ones [28]. Their subsequent experimental study found no overall effect of graphic type on risk estimates or risk feelings [27]. While our study did not ask participants about how the graphics made them feel about their risk, we did find that people were less likely to choose the clearly optimal option after having used an interactive risk graphic than they were with a standard risk graphic. This suggests that the increased emotional response observed by Ancker et al may be an artifact of their game-like task and that more straightforward types of interactivity may not lead to improved choices.

Research on other forms of interactivity in health education such as video games [38] and immersive 3-dimensional environments [39] has suggested caution in considering how these applications might or might not improve outcomes in health education. Our results complement these prior findings by showing that interactive approaches to risk communication should come with similar caveats.

Interactive risk graphics are intriguing, but their use may be counterproductive to the purpose of effectively communicating to patients about health risks. While the movement of patient decision aids from booklet and DVD formats to online resources has created opportunities for greater technological sophistication, our research should serve as a cautionary tale to developers and educators seeking to ensure that their materials are “cutting edge.” Our teach-back interactive graphing task was intended to reinforce deeper cognitive processing of decision-relevant risk information, exactly the type of thinking most decision aids seek to promote in patients. Yet, in this case, our intervention backfired and created worse decision outcomes. More research is clearly needed to evaluate different types of interactive risk communications and to identify the design features of interactive exercises that lead to better results versus the features of those that do not. In the meantime, decision-aid designers should proceed with caution when considering the use of flashy risk graphics.