Ethical approval for the study was obtained through the Human Research Ethics Committee of the Sydney Local Health District (Protocol No. X11-0200). Each participant gave written consent before participating in the interview.

General practitioners (GPs) in New South Wales, Australia, recruited 26 patients between 40-70 years of age, with at least one CVD or lifestyle risk factor, who were not currently taking medication, targeting low (5-year absolute risk<10%) to moderate (10-15%) risk patients who may be less motivated by their percentage risk result [20,21]. Purposive sampling was used to recruit participants with a range of ages, gender, knowledge of risk factors, and risk calculator results (see Table 1). This was done by modifying the eligibility criteria given to recruiting GPs throughout the study. Analyses based on 26 participants suggested saturation of key themes (see Figure 1), so no further recruitment was conducted [22].

Participants were 16 women and 10 men, between 40 and 67 years of age, with a range of highest educational attainment: 4 had not completed high school, 6 had completed high school, 7 had a technical qualification, and 9 had completed a university degree. Five-year absolute risk results ranged from 1-12%, with 23 at low risk (<10%) and 3 at moderate risk (10-15%) of a CVD event. Compared to current age, the heart age results were: 16 older, 4 younger, 1 same as current age, and 5 mixed results for the two calculators.

Two heart age calculators based on Framingham risk equations were publicly available at the time of the study in 2012 (see Table 2 and Figure 2) [23,24]. A protocol including think aloud and semi-structured interview methods was developed based on past research showing that a concurrent think aloud protocol elicits more information, but additional insights can be gained retrospectively (see Multimedia Appendix 1 for protocol) [25]. The interviewer (CB) was a researcher trained in public health qualitative methods, who piloted the protocol with a convenience sample of 4 participants who met the study eligibility criteria. Pilot participant feedback was discussed with the research team and the think aloud instructions were clarified before commencing the study. Participants were asked to think aloud as they used each website in random order, with minimal input from the interviewer unless they had difficulty using the website. No interpretation of the results was provided until the end of the interview and the interviewer clarified that the researchers were not connected to the websites if this issue arose. In order to practice thinking aloud consistently, participants completed a simple “spot the difference” task in which they described what they were doing. Upon successful completion of this practice task, participants began using the heart age calculators. A “keep talking” sign was placed above the computer, which the interviewer would point to if the participant was silent for more than 10 seconds. The entire session was audio-recorded and use of the websites was video-recorded using SMRecorder screen capture software [26]. All interviews were audio-recorded, but technical problems prevented the use of screen capture software in some interviews: 3 participants had no video data because of computer or software issues and 5 participants had only one calculator video-recorded (3 due to computer or software issues and 2 due to the Unilever website being taken down, who completed the study over the phone when the website became temporarily available again).

A framework analysis method was used to analyze the interview transcripts, which involved five steps [27]. The first step was familiarization with the data: CB read through all 26 transcripts, recorded the calculator input/output for each participant from videos if available or from transcripts and field notes if unavailable, and discussed this and 2 transcript excerpts with all authors, covering younger and older heart age results. The second step was to create a thematic framework: CB, JJ, and KM read a sample of 5 transcripts covering younger, same/mixed, and older heart age results, and developed the initial framework. Third was indexing: CB, JJ, and HD each watched a video to ensure understanding of the process and coded the remaining transcripts according to the framework, with new themes and revisions to the framework discussed (see Multimedia Appendix 2 for final framework). Ten transcripts were double-coded independently by two researchers. The fourth step in the analysis was charting: CB, JJ, and HD summarized themes and supporting quotes from each transcript in the framework (a matrix with participants as rows and themes as columns), with transcripts reread and discussed to resolve any disagreement about the best way to represent the data. The fifth step was mapping and interpretation: CB and JJ examined the framework within and across themes and participants to identify overarching themes and relationships, independently summarized the process of using and interpreting the two risk calculators in a diagram, and differences between the 2 process diagrams were discussed with KM and HD. The order in which the calculators were viewed was taken into account, based on separate coding of the two websites, but this did not appear to influence the overall process. Then JJ, HD, and KM each read 2 additional transcripts to check the findings and the final process diagram was created (see Figure 1). The final results were discussed with all authors. Rigor was addressed by: repeated coding of transcripts by different team members to ensure a comprehensive themes list and framework was achieved; an iterative process of constant comparison between the existing framework and new data; detailed documentation of the analysis process; and discussion of emerging and final themes with all authors [28].

The reliability of the risk calculator results was affected by several factors. Misinterpretation of risk factor questions was common, with 9/26 participants making at least one error in their responses to the questions, based on inconsistencies between the data entered in the two calculators and their thought process while entering the data. Many participants couldn’t remember their exact blood pressure or cholesterol levels but were aware that these risk factors were low or normal based on past assessment, and this did not match the estimates provided by the calculator.

The two calculators gave different heart age results for all but one participant due to these input errors and the different assumptions built into the calculators: the use of different absolute risk models (5- vs 10-year risk; use of body mass index if cholesterol was unknown), the estimates used when blood pressure was unknown, and the “ideal” that participants were compared to when calculating the heart age.

In the New Zealand calculator, the explanation of the percentage risk information was often overlooked, with 12/26 participants skipping through at least one part of the explanation. Even when the percentage information was seen, it was often forgotten by the end, with participants generally more focused on the graph and heart age results. When specifically asked, many participants were unsure what the percentage referred to, even though the graph title clearly stated it was the percentage risk of a heart attack or stroke in the next 5 years.

The graph in the New Zealand calculator was sometimes confusing, particularly the first few screens where many different numbers were used in the explanation of the results (see Figure 2A). However, other participants liked the graph and demonstrated a clear understanding of how risk would increase with age when viewing it.

The heart age format in both calculators was much more salient, meaningful, and elicited more emotional reactions in participants. Younger heart age was seen as very positive and participants often preferred the calculator that gave them a lower heart age, although some felt it was unrealistic and could discourage people from improving their lifestyle. Older heart age was confronting and participants’ first reaction was often disbelief, particularly if they felt they had a good lifestyle. Heart age had a more immediate meaning to participants than percentage risk, indicating a healthy lifestyle if younger, and the need to change if older.

The multiple formats presented in the New Zealand calculator were sometimes perceived as contradictory, particularly low percentage risk in the green mild category compared to an older heart age result. The use of consistent colors was also important—green indicated a mild risk level in the basic graph, but a green line was also used to show how risk would increase if the participant started smoking or developed diabetes.

The ability to modify different risk factors to see what effect they had on the results also had mixed responses from participants: some weren’t interested in using it or didn’t understand it, while others spent some time playing around with the factors to reduce their heart age result or make the estimated risk factors more realistic. However, the message this conveyed depended on how the participant used it—for example, the following participant concluded that blood pressure was more important than cholesterol because she happened to move blood pressure to a higher level.

Participants tended to focus on the risk factors that were most relevant to them; for example, the effect of smoking was of more interest to current/ex-smokers and several participants wanted to see the effect of alcohol.

The process of using the risk calculators involved several stages: expectations of CVD risk based on prior knowledge, experience of using the calculator, evaluation of the credibility of the results, and actions considered as a result of this process. This is illustrated in Figure 1, with examples in Tables 3 and 4. Participants’ expectations, experience, evaluation, and actions sometimes changed between the two calculators, but the order in which they were viewed did not appear to affect the overall process. The process diagram therefore describes the range of pathways that participants followed regardless of the order in which the calculators were used, with solid lines indicating the two main pathways, and dashed lines indicating alternative pathways.

The high credibility pathway tended to occur when participants had little prior knowledge of their own risk of CVD. Their general knowledge of CVD risk factors was consistent with the information in the calculators, and so they tended to accept the results and have a more positive reaction to them. Those who received a younger heart age result usually had a positive reaction. In these situations the credibility of the results was not closely questioned and led to various actions: participants considered changing their lifestyle to lower risk even further or maintain younger heart age, thought of higher risk family or friends who could benefit from using the calculator, and sometimes perceived increased understanding or changed risk perception.

Alternatively, those with more specific knowledge evaluated the calculator as having high credibility if they had a positive reaction to the calculator, such as getting lower heart age than current age or all the information matched what they knew about their CVD risk. Seeing a similar result for the second calculator also increased credibility.

The low credibility pathway tended to occur when participants had more specific knowledge of their own risk of CVD, but could not necessarily remember their exact blood pressure and cholesterol levels. These participants were more likely to encounter unexpected information in the calculators, and reacted negatively to receiving an unexpected heart age result. In these situations, the credibility of the results was questioned and participants re-evaluated their prior expectations (eg, that they already had a good diet) and experience of using the calculator (eg, that they weren’t asked any questions about diet) to make sense of the result. Common reasons used to discredit the results were the lack of relevant lifestyle questions in the New Zealand calculator and the influence of corporate sponsorship in the Unilever calculator. However, even when the results were rejected, participants still considered lifestyle change and felt that the calculators would be useful for others. Some decided they should get their blood pressure or cholesterol checked again to increase the accuracy of the risk assessment.

Alternatively, those with little prior knowledge evaluated the calculator as having low credibility if they had a very negative reaction to the results or specific components (eg, some men believed that body mass index was inaccurate for them). However, they still considered lifestyle change and getting a more accurate assessment.

In summary, using the heart age calculators appeared to lead participants to consider lifestyle changes regardless of the pathway they described and regardless of their heart age result. This is illustrated by the quotes in Tables 3 and 4.

Our findings suggest that online heart age calculators prompt people to consider improving their lifestyle regardless of the accuracy and perceived credibility of the results, or the result they received. This is consistent with the findings of the “lung age” study, where both positive and negative results prompted smokers to consider quitting [18]. As found in previous studies on diabetes and cancer risk calculators, participants often disregarded unexpected or negative information [1,5], and actively sought reasons to discredit the results. However, participants in this study considered lifestyle change and felt the calculators would be useful for other people, even when they claimed to disbelieve the calculator result. The value of such tools is therefore dependent on their goal—if it is to prompt people to think about improving their lifestyle regardless of the level of risk [4], they appear to be achieving this; if it is to provide accurate information and understanding of risk, then they could be improved [1]. However, thinking about lifestyle changes does not necessarily translate into actual behavior change. Previous research has found that personalized risk calculators have limited impact on behavior [29], but they may prompt people to seek further information and support as an interim step before behavior change [30].

The ultimate goal of risk calculator websites will vary depending on the motivation and target audience of the organization that created it. Our findings show wide variation in the way that people use and understand features of personalized risk calculators, with implications for designers of such tools. Patients in our study misinterpreted risk factor questions, entered data inconsistently between different risk calculators, skipped information that may have prevented these issues, and did not use all features of the tools. Some of these issues may be improved by simple design alterations, such as larger font, easier navigation, and clearer instructions for how to use the risk factor modification tools. Alternatively, simpler, less interactive information formats may be more effective in terms of information processing and understanding of risk [31,32].

This study is consistent with previous findings that heart age elicits more emotional reactions and is more meaningful to patients than percentages [16,19]. All participants demonstrated good understanding of the effect of risk factors on heart age and its link to lifestyle, but the percentage information was interpreted in many different ways and often overlooked. This supports the large body of research showing that percentages are often misunderstood [10,11]. The downside of heart age was that it was very confronting for participants to receive a much older heart age than their current age and some explicitly said they would prefer not to know.

The presentation of multiple formats was problematic because low percentage risk and older heart age were perceived as contradictory, suggesting that the focus should be on explaining one or the other, not both. Although the New Zealand calculator was developed with a step-wise structure that attempts to fully explain how the results were calculated [33], the large amount of information with multiple numbers was confusing for many participants, and the simplified Unilever format was often preferred even though corporate sponsorship reduced its credibility. The graph used in the New Zealand calculator with color-coded risk categories and projected risk over age (see Figure 2) appeared to be useful additional information for many but not all participants. Since preferences for and understanding of different CVD risk formats were variable in our study, quantitative research is needed to test the effect of presenting heart age in different formats and identify the best way to present such information to different groups of people. Future research should include measurement of benefits like understanding and motivation to improve lifestyle, but also the potential harms of conveying such emotive information, including worry and seeking unnecessary tests for low risk.

Future research on online health risk calculators could also investigate how to increase their reliability and credibility. One option is to involve clinicians in explaining the results, which could improve perceived credibility and the accuracy of the risk factor data, and may prevent misunderstandings. However, since health consumers use online risk calculators outside of clinical consultations [1], it would be beneficial to improve the format of online health information so that unexpected or absent information is fully explained. This should include an explanation of why different risk calculators may produce different results, to increase awareness of accurate data entry for risk factors, and understanding of the assumptions behind the calculation [2]. Our findings suggest that people expect to be asked about broader aspects of their lifestyle and history than those included in the CVD risk models; consequently, the face validity is reduced by exclusion of these questions. The link between lifestyle advice and the risk factor questions also needs to be clear to avoid a negative reaction to the calculator. However, the aim of risk calculator tools will vary depending on the goals of the organization that develops them and the audience they are targeting, so our findings may have different implications for different designers.

The strengths of this study include a novel topic and rigorous qualitative methods, including purposive sampling to achieve a heterogeneous patient sample, theme saturation, a trained interviewer, use of both semi-structured interview and think aloud data, and a framework analysis process that used multiple analysts and an iterative process to arrive at final themes. The external validity of the study is strengthened by the use of existing online risk calculators, self-reported risk factors, and widely used, validated Framingham risk equations. The limitations include missing video data for some participants and the possibility that the interview questions and presence of the interviewer may have influenced reactions to the websites. However, audio recordings were obtained for all participants, most users of such websites would have a prior interest in CVD, and the interviewer took care to avoid giving any interpretation of the results or reactions to the websites. The results may not reflect how consumers use risk calculators in a more realistic setting, and as typical with qualitative research, the sample was not designed to be representative of the general population but rather present a range of perspectives.

Our findings demonstrate an interesting paradox: online heart age calculators are easily misunderstood and the results may be dismissed if the information is unexpected or negative, but the process of using such calculators may motivate lifestyle change regardless of the outcome. Future research should investigate both the benefits and harms of communicating risk in this way and how to increase the reliability and credibility of online health risk calculators.