The Karma system is currently undergoing maintenance (Monday, January 29, 2018).
The maintenance period has been extended to 8PM EST.

Karma Credits will not be available for redeeming during maintenance.

Journal of Medical Internet Research

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 01.02.18 in Vol 20, No 2 (2018): February

This paper is in the following e-collection/theme issue:

    Review

    Clinical Validity, Understandability, and Actionability of Online Cardiovascular Disease Risk Calculators: Systematic Review

    1School of Public Health, The University of Sydney, Camperdown, Australia

    2Ask, Share, Know: Rapid Evidence for General Practice Decisions Centre for Research Excellence, Discipline of General Practice, The University of Sydney, Camperdown, Australia

    Corresponding Author:

    Carissa Bonner, MPH, PhD

    School of Public Health

    The University of Sydney

    Edward Ford Building (A27) Room 226A

    Camperdown,

    Australia

    Phone: 61 2 9351 7125

    Email:


    ABSTRACT

    Background: Online health information is particularly important for cardiovascular disease (CVD) prevention, where lifestyle changes are recommended until risk becomes high enough to warrant pharmacological intervention. Online information is abundant, but the quality is often poor and many people do not have adequate health literacy to access, understand, and use it effectively.

    Objective: This project aimed to review and evaluate the suitability of online CVD risk calculators for use by low health literate consumers in terms of clinical validity, understandability, and actionability.

    Methods: This systematic review of public websites from August to November 2016 used evaluation of clinical validity based on a high-risk patient profile and assessment of understandability and actionability using Patient Education Material Evaluation Tool for Print Materials.

    Results: A total of 67 unique webpages and 73 unique CVD risk calculators were identified. The same high-risk patient profile produced widely variable CVD risk estimates, ranging from as little as 3% to as high as a 43% risk of a CVD event over the next 10 years. One-quarter (25%) of risk calculators did not specify what model these estimates were based on. The most common clinical model was Framingham (44%), and most calculators (77%) provided a 10-year CVD risk estimate. The calculators scored moderately on understandability (mean score 64%) and poorly on actionability (mean score 19%). The absolute percentage risk was stated in most (but not all) calculators (79%), and only 18% included graphical formats consistent with recommended risk communication guidelines.

    Conclusions: There is a plethora of online CVD risk calculators available, but they are not readily understandable and their actionability is poor. Entering the same clinical information produces widely varying results with little explanation. Developers need to address actionability as well as clinical validity and understandability to improve usefulness to consumers with low health literacy.

    J Med Internet Res 2018;20(2):e29

    doi:10.2196/jmir.8538

    KEYWORDS



    Introduction

    Online health information may be the first step toward seeking professional medical advice, so the quality of this information is important: is it clinically valid, does it communicate risk effectively, is it understandable to the user, and what actions does it prompt? Unfortunately, the majority of users may not have the necessary skills to effectively evaluate these issues. Health literacy is the ability to access, understand, and make use of health information and services [1], and a large proportion of the general population has inadequate skills [2] (poor health literacy skills in Australia: 59% [3], Europe: 47% [4], Canada: 60% [5]). Low health literacy is associated with less trust in online health information, decreased ability to evaluate that information, and worse health outcomes for cardiovascular disease (CVD) and other chronic conditions requiring self-management [2,6,7]. The issue of eHealth literacy is a related but separate barrier to using health information online—not only do users need to understand and act on the information, but they need the basic skills to find reliable websites in the first place [8]. Less educated, low income, and older individuals may be particularly disadvantaged by inaccessible and poorly explained online health information [2,6,9]. While there is no consensus on how best to evaluate health websites, the majority of studies have concluded that the quality is low [10]. EHealth interventions are increasingly common but have largely neglected the issue of health literacy, which may contribute to their low use [7,8,11].

    CVD is the leading cause of mortality and morbidity worldwide, but its incidence can be reduced through risk factor modification via lifestyle change and/or medication [12-15]. This makes it a highly relevant issue for eHealth, as it affects a large proportion of the population and may be prevented through individual behavior change before medical intervention is necessary. Many eHealth interventions target lifestyle with the ultimate aim of preventing CVD [11], but how do individuals know when to access professional medical assistance? The decision to prescribe medication should be based on the likelihood of avoiding a heart attack or stroke, which depends on the baseline absolute CVD risk for an individual [16]. CVD prevention guidelines often use algorithms based on large cohort studies to estimate the risk of a CVD event, usually over 5 or 10 years [12,13]. For example, by identifying 1000 patients with an absolute CVD risk of >21% and lowering their blood pressure, 38 heart attacks and/or strokes would be prevented over 5 years [17,18]. On the other hand, identifying 1000 patients with an absolute CVD risk <11% and treating their blood pressure would prevent only 14 CVD events over 5 years [17,18]. Both groups would however, be exposed to the potential side effects, costs, and inconvenience of antihypertensive medication in order to achieve these reductions in CVD event rates [18]. To estimate the absolute risk of a CVD event, numerous tools exist using different parameters and models [19]. The commonly used Framingham model is based on age, sex, smoking, diabetes, cholesterol, and blood pressure [20]. More recent models used in UK and US guidelines include ethnicity and socioeconomic indicators [13,15]. These risk calculators are available to the public online, but little is known about their quality.

    Previous research indicates that online CVD risk calculators can be easily misunderstood. Users may enter their risk factors incorrectly, the provision of multiple risk formats can be confusing if not explicitly explained, and the risk calculators themselves may make assumptions about missing data that lead to less accurate results [21,22]. Users may also question the credibility of the calculator’s results if their prior expectations are not met [23]. On the other hand, an engaging interactive format can increase the emotional response to the risk result and potentially motivate action more than a standard verbal description of risk by a doctor [24,25].

    This study aimed to systematically review publicly available online risk calculators for CVD and evaluate them on criteria relevant to health literacy (clinical validity, risk communication, understandability, and actionability).


    Methods

    Procedures

    The general approach for this study was to follow a systematic review process using 2 independent searchers (SH, RS) who qualitatively described each risk calculator and evaluated them quantitatively based on the validated Patient Education Material Evaluation Tool for Print Materials (PEMAT-P) scale. For the search and evaluation, we used a third rater (MF) to resolve discrepancies and reach consensus in accordance with section 7.6 of the Cochrane Handbook for Systematic Review of Interventions [26]. The qualitative descriptions of the risk calculator were used to develop a framework for quantitative data extraction (risk model, risk result, and presence/absence of risk communication formats), after which an individual researcher (MF) conducted the basic data extraction. This process was discussed and refined with the lead researcher (CB) on a fortnightly basis, with additional advice from a general practitioner/academic researcher (LT).

    Ethical Approval

    Since there were no participants in this study and the data was based on publicly available websites, an ethics application was not required.

    Inclusion and Exclusion Criteria

    Online risk calculators were considered if they met the following inclusion criteria: (1) assessed risk of future CVD in individuals without a previous CVD event, (2) were available without the need for registration or payment, and (3) were interactive. Calculators were not considered if they were downloadable files such as an Excel (Microsoft Corp) spreadsheet or PDF, addressed absolute risk of future cardiovascular events in people with atrial fibrillation, or did not provide a risk result for the end user.

    Search Strategy

    There were 2 main search strategies for identifying Web addresses that contained the CVD risk calculators. The first strategy was to access predetermined reputable websites including 6 national heart foundation websites (Australian National Vascular Disease Prevention Alliance, the National Heart Foundation of New Zealand, the Joint British Societies, the UK National Health Service, the American Heart Association, and the American College of Cardiology) and a not-for-profit source [27], and the second strategy used Google Australia with English-language terms. The 2 independent searchers (RS, SH) were instructed to reset their cache in their Web browsers before each Google search to minimize the effect of Google search optimization. The 2 search term themes were “CVD/medication” and “risk.” The lead researchers (CB, LT) and the 2 independent searchers agreed upon 11 specific terms for CVD/medication (CVD, heart disease, stroke, heart attack, hypertension, hypercholesterolemia, hypercholesterolaemia, aspirin, blood pressure medication, cholesterol medication, and statin) and 2 specific terms for risk (risk calculator and risk assessment). A single CVD/medication and a single risk term were combined for a single search resulting in 22 unique Google searches. The first 50 results were considered (not including Web advertisements), providing a pool of 1100 results to be title scanned. The search results were limited to the first 50 after method piloting showed no additional websites would have been included up to 100 results. Searchers only recorded the Web addresses if they were to be assessed for eligibility. Duplicates were considered either as identical Web addresses, Web addresses that linked to the same risk calculator, or where a risk calculator from one webpage was embedded in another.

    The searchers (SH, RS) conducted this search as part of a Master of Public Health degree capstone unit from August to November 2016. In March 2017, an independent member of the team (MF) reconciled these search results based on the record of screened Web addresses provided by the original searchers (see Figure 1) by using Excel and removing duplicate Web addresses. The 2 independent searchers then rated risk calculators with the PEMAT-P providing 2 PEMAT-P ratings for 1 risk calculation.

    Figure 1. Search strategy and results (updated with higher res image).
    View this figure

    Evaluation and Data Extraction

    The 2 searchers (SH, RS) developed the framework for basic data extraction by qualitatively describing the content of the risk calculator results for the high-risk profile. From this, a standard form was developed (CB, MF) to numerically record basic descriptive data for each calculator: risk model used, how the risk was presented (eg, relative risk, absolute risk, life expectancy, graphical formats) and recommended actions for the high-risk profile (eg, take medication, change lifestyle, see general practitioner). The third rater (MF) then extracted the data numerically, with any uncertainties discussed with the lead researcher (CB) to reach consensus.

    The 2 searchers (SH, RS) also rated the content of each risk calculator using a validated tool, the PEMAT-P [27]. Searchers were instructed to first read the PEMAT-P user manual before proceeding with rating and spoke fortnightly with a supervising researcher (CB) to discuss/explain any items that were not immediately clear (eg, examples of an active voice). PEMAT-P provides 2 submeasures that are particularly relevant to health literacy: understandability, which is a measure of how well a health consumer is able to process and explain the key message of the material, where higher percentages indicate better understandability, and actionability, which is a measure of how well a health consumer is able to identify what to do based on the information presented, where higher percentages indicate better actionability. For the 2 independent searchers, the correlation between understandability scores was 0.57 and the correlation between actionability scores was 0.71. Discrepant item scores between the first 2 raters were resolved by a third rater (MF, after discussion with CB) to finalize the PEMAT-P score for each risk calculator. These decisions were double-checked with the original searchers, who agreed with the final approach.

    A predefined high-risk cardiovascular profile for a hypothetical patient was used to assess the clinical validity of each calculator. This was a 65-year-old male smoker with systolic/diastolic blood pressure of 130/80 mm Hg, total/high-density lipoprotein cholesterol ratio of 6, and body mass index of 26 kg/m2. Where risk calculators had additional factors, the question was left blank (if possible) or an answer was given that either indicated the middle of the range or provided no additional risk on top of the risk profile (eg, a low-risk ethnicity, no history of CVD or taking medication). The 2 independent raters (SH, RS) created descriptive lists of the different clinical risk models, risk formats, and recommended actions for each calculator based on the high-risk profile. The third rater (MF) then used this framework to code the full dataset after discussion with CB, using the same high-risk profile.


    Results

    Search Results

    This search yielded 67 unique webpages (see Figure 1). A list of the included Web addresses and their ID numbers can be found in Multimedia Appendix 1 (Table A). One website would sometimes host multiple risk calculators for different CVD events (eg, specifically for stroke or myocardial infarction). In total, these 67 webpages were found within 56 websites. Three calculators were able to calculate multiple absolute risks of future CVD events based on different models (ID2, ID4, ID10). These calculators were counted only once in the search strategy but are duplicated for the purposes of data extraction as different models of future CVD risk use different risk factors. Risk factor profiles are based on model-specific results. For example, ID2 is able to calculate risks based on 3 different models, therefore has been counted as 3 unique calculators. From the included 67 webpages, there are a total of 73 unique risk calculators.

    Risk Calculator Characteristics

    The descriptive characteristics of the calculators are provided in Table 1.

    The calculators used a variety of published risk models but the most common were those used in clinical practice guidelines: Framingham (44%), American College of Cardiology/American Heart Association (ACC/AHA) (10%), QRISK2 (5%), Reynolds (4%), and Assessing cardiovascular risk using the Scottish Intercollegiate Guidelines Network (ASSIGN) (3%). One-quarter did not specify the underlying model (25%). The outcomes included CVD/coronary heart disease (CHD), angina, heart attack/myocardial infarction, stroke, heart failure, kidney disease, diabetes, and CVD/CHD death. Most calculators used a 10-year time frame (77%) but some used 5-year (7%), 30-year (3%), or lifetime (7%) risk, and one allowed different estimates for the range of 1 to 10 years (1%). Many risk calculators did not state specific outcomes beyond mentioning CVD risk, and some did not state the time frame.

    The absolute percentage risk was stated in most but not all calculators (79%). Other risk formats included categorical risk with 2 to 4 groups ranging from low to high (32%), a verbal description of the frequency such as 8 in 100 (18%), heart age (10%), life expectancy (3%), and relative risk (3%). In 34 risk calculators (47%) only 1 risk format was presented, and in 39 (53%) risk calculators 2 to 5 risk formats were presented. The risk profile outlined previously yielded highly variable results depending on the model and outcomes used, with an absolute risk ranging from 3% to 43% over 10 years, a heart age of 68 to 86 years, a life expectancy of 79 to 84 years, and a relative risk of 1.8 to 2.1 compared to a healthy person’s risk. Visual aids included icon arrays or pictographs (18%), bar or line graphs (16%), and charts showing risk level (10%).

    For the PEMAT-P evaluation, the calculators scored moderately on understandability and poorly on actionability. The average understandability score was 64% (SD 20%) which ranged from 30% to 100%, and the average actionability score was 19% (SD 26%) which ranged from 0 to 100%. Screenshots from very high-scoring examples for understandability (ID14) and actionability (ID66) are shown in Figures 2 and 3, respectively. Reliability was variable for the individual PEMAT-P items, with Cohen kappa scores ranging from –0.05 to .65 and agreement ranging from 42% to 99%. These discrepancies were resolved by using a third rater and team discussion to reach consensus on each decision. The PEMAT-P scores for individual risk calculators and reliability by PEMAT-P item are provided in Multimedia Appendix 1 (Tables B and C).

    Table 1. Characteristics of final calculators (n=73).
    View this table
    Figure 2. An example of a risk calculator with a high understandability Patient Education Material Evaluation Tool for Print Materials score.
    View this figure
    Figure 3. An example of a risk calculator with a high actionability Patient Education Material Evaluation Tool for Print Materials score.
    View this figure

    Discussion

    Principal Findings

    This review found 73 unique CVD risk calculators available to the public online that use a wide variety of risk models, risk communication formats, understandability, and actionability. Of particular concern is the variation in CVD risk estimates based on entering the same hypothetical high-risk patient data with little explanation for why this would occur: ranging from as little as 3% to as high as a 43% risk of a CVD event over the next 10 years. One-quarter (25%) of the risk calculators did not specify the underlying model, study, calculations, or assumptions, so it not possible to assess their validity or the reasons for variation. The remaining three-quarters (75%) did specify enough information to determine the underlying algorithm, where differences in the study population (eg, US-based Framingham versus UK-based QRISK), included risk factors (eg, whether less directly predictive factors like body mass index were included as well as strong clinical indicators like cholesterol), and CVD outcomes (eg, mortality only, nonfatal heart attack and stroke, angina) explain the discrepant risk results for the same profile. In the academic literature, there are over 17 validated CVD risk models [19], and both the United Kingdom and United States have moved on from the original 10-year Framingham model in their current clinical guidelines [13,15]. In contrast, Framingham is still the dominant model online, but there were no publicly available 5-year risk calculators that scored very highly on both understandability and actionability. This means that countries like Australia that use a 5-year Framingham model could benefit from additional development using highly rated features of 10-year calculators (Figures 2 and 3). Patients with low health literacy searching for their own CVD risk information are most likely to encounter 10-year models that do not correspond to local clinical guidelines.

    In terms of risk communication, absolute risk should ideally be presented in a variety of formats to cater to different needs and learning styles [28]. While absolute risk, verbal explanations of risk categories, and graphical presentations were all present in our sample of risk calculators, just over half used a combination of different presentation formats for the same numerical information. At minimum, all calculators should present the absolute risk in numerical form, but over 20% did not meet this minimum standard. Icon arrays displaying risk in terms of CVD event frequencies were only used in 18% of risk calculators, despite much attention in the risk communication literature and recommendations to use these formats in international patient decision aid standards [28].

    The PEMAT-P evaluation was chosen for its unique focus on reducing health literacy demand through both understandability and actionability [29]. It does not yet have an agreed threshold for acceptable levels, but comparison to recent studies of information for related conditions suggests that understandability was moderate and actionability was poor overall (present study: 64% and 19%; online heart failure websites: 56% and 35%; printed lifestyle information for chronic kidney disease: 52% and 37% for understandability and actionability, respectively) [30,31]. To improve actionability, clinical guidelines recommend both lifestyle and medication for the selected high-risk patient profile [12], but even the most strongly recommended action of quitting smoking was only mentioned by 29% of calculators, and there was more focus on statins than blood pressure–lowering medication even though both risk factors were elevated compared to ideal levels. Presenting clear, jargon-free information on all the available options as well as referring the user to a doctor to discuss high-risk results is recommended to better meet the needs of low health literacy users.

    The findings of this study are comparable to broader literature on the quality of online health information; a review showing 70% of studies evaluating 5941 websites concluded that higher quality is needed [10]. For a recent example, the US Department of Health and Human Services’ Office of Disease Prevention and Health Promotion evaluated the quality of the 100 top-ranked health-related websites and found that only 58% met at least 3 out of 6 reliability criteria, while 42% followed at least 10 out of 19 usability principles [32]. This was part of an effort to set national objectives to improve the quality of eHealth by 2020, to which health literacy criteria could perhaps be added in future.

    Strengths and Limitations

    The strengths of this study include a systematic review and evaluation process with multiple independent searchers/raters. The main limitation is the replicability of conducting a systematic search using online search engines like Google. The dynamic nature of the Web with constant variation in website content and metadata means that no search is perfectly replicable even though the cache was cleared between search terms. However, the methods used are likely to have captured the most common and popular search results, since many duplicates were removed between the 2 searchers. It is likely that additional calculators existed at the time of the search and could potentially have been found by a different searcher, search engine, or geographical location, but this study provides a comprehensive list of accessible calculators at the time of searching.

    The variable reliability of the PEMAT-P items was slightly lower than previous research [33], but could possibly be improved by recoding the low-scoring items or through further training of the coders, although the PEMAT-P developers intended it for use by nonexperts [29]. Lower PEMAT-P reliability could also be due to the risk calculators not fitting PEMAT definitions for printable materials. Reevaluation was not possible in this study due to the changing nature of the interactive risk calculator websites but could be considered for future research using PEMAT-P with static materials. The data extraction for basic descriptive content (risk model and risk result formats) was conducted by an individual researcher in this study, so reliability could not be assessed, although this is not generally reported in systematic review data extraction [26].

    Implications

    The plethora of calculators and wide variation in results from the same input have the potential to confuse and harm the general public if appropriate medical advice is not sought. Actionability scores are poor on average, and minimum risk communication standards are not being met. Future research evaluating the suitability of online risk calculators for low health literacy users would benefit from a revised version of the PEMAT-P designed specifically for interactive online formats. Existing items relevant to health literacy may need to be clarified, including (1) better definition for the presence/absence of a table, since website content is often built around a table format, (2) instructions for dealing with interactive graphs based on buttons, (3) a definition of short material for webpages, (4) how to address subtle visual aids such as color, size, or positioning which are more heterogeneous in interactive online formats, and (5) how to best address actionability items with general health advice versus personalized advice based on the risk result. In addition, concepts from eHealth evaluation tools could be incorporated, including (1) ease of navigation through the calculator; (2) presence/absence of distractions in the webpage (eg, advertisements, pop-ups); and (3) ability to save, print, or email the results and recommended actions. Finally, additional information is needed to enable clinicians and consumers to determine whether health risk calculators give a reliable estimate, including the model or algorithm used (eg, study name/reference, outcomes measured, and time frame), what population the calculation has been validated in (eg, age, sex, and ethnicity), an explanation of how this relates to current clinical guidelines, and when the evidence for the calculation was last updated.

    Conclusion

    Online CVD risk calculators produce highly variable results for the same person with little explanation for why this would occur. Differences in the models used, risk factors included, risk communication formats presented, and actions specified explain part of this variation, but one-quarter of risk calculators did not specify any underlying assumptions. Health professionals should be aware of the reasons for conflicting results that patients might encounter, and developers need to address actionability as well as clinical validity and understandability to improve usefulness to the majority of the population with low health literacy. Country-specific calculators that match national clinical guidelines and build on examples with high understandability and actionability scores would benefit both health professionals and consumers.

    Acknowledgments

    The study was supported by the National Health and Medical Research Council of Australia through the Ask, Share, Know: Rapid Evidence for General Practice Decisions Centre for Research Excellence. CB contributed to the conceptualization, methodology, data analysis and interpretation, and drafting the manuscript. MF contributed to the methodology, data analysis, and revising the manuscript. SH contributed to the methodology, data collection, and revising the manuscript. RS contributed to the methodology, data collection, and revising the manuscript. LT contributed to the conceptualization, methodology, and revising the manuscript.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Risk calculator archive and Patient Education Material Evaluation Tool for Print Materials scores.

    PDF File (Adobe PDF File), 64KB

    References

    1. Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promotion International 2000;15(3):259-267. [CrossRef]
    2. Adams RJ, Appleton SL, Hill CL, Dodd M, Findlay C, Wilson DH. Risks associated with low functional health literacy in an Australian population. Med J Aust 2009 Nov 16;191(10):530-534. [Medline]
    3. Adult literacy and life skills survey, summary results. Australia (reissue). Sydney: Australian Bureau of Statistics; 2006.   URL: http:/​/www.​ausstats.abs.gov.au/​ausstats/​subscriber.nsf/​0/​B22A471C221C7BADCA2573CA00207F10/​%24File/​42280_2006%20(reissue).​pdf [accessed 2017-07-24] [WebCite Cache]
    4. Sørensen K, Pelikan JM, Röthlin F, Ganahl K, Slonska Z, Doyle G, et al. Health literacy in Europe: comparative results of the European health literacy survey (HLS-EU). Eur J Public Health 2015 Dec;25(6):1053-1058 [FREE Full text] [CrossRef] [Medline]
    5. Rootman I, Gordon-El-Bihbety D. A vision for a health literate Canada: report of the Expert Panel on Health Literacy. Ontario: Canadian Public Health Association   URL: https://www.cpha.ca/vision-health-literate-canada-report-expert-panel-health-literacy [accessed 2017-07-31] [WebCite Cache]
    6. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011 Jul 19;155(2):97-107. [CrossRef] [Medline]
    7. Diviani N, van den Putte B, Giani S, van Weert JC. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res 2015;17(5):e112 [FREE Full text] [CrossRef] [Medline]
    8. Mackert M, Champlin SE, Holton A, Muñoz II, Damásio MJ. eHealth and health literacy: a research methodology review. J Comput-Mediat Comm 2014 Apr 12;19(3):516-528. [CrossRef]
    9. Watkins I, Xie B. eHealth literacy interventions for older adults: a systematic review of the literature. J Med Internet Res 2014;16(11):e225 [FREE Full text] [CrossRef] [Medline]
    10. Eysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the quality of health information for consumers on the World Wide Web. JAMA 2002 May 22;287(20):2691. [CrossRef]
    11. Kohl LFM, Crutzen R, de Vries NK. Online prevention aimed at lifestyle behaviors: a systematic review of reviews. J Med Internet Res 2013;15(7):71-83 [FREE Full text] [CrossRef] [Medline]
    12. Guidelines for the management of absolute cardiovascular disease risk.: National Vascular Disease Prevention Alliance; 2012.   URL: https://www.heartfoundation.org.au/images/uploads/publications/Absolute-CVD-Risk-Full-Guidelines.pdf [accessed 2018-01-10] [WebCite Cache]
    13. JBS3 Board. Joint British Societies' consensus recommendations for the prevention of cardiovascular disease (JBS3). Heart 2014 Apr;100 Suppl 2:ii1-ii67. [CrossRef] [Medline]
    14. Lim SS, Vos T, Flaxman AD, Danaei G, Shibuya K, Adair-Rohani H, et al. A comparative risk assessment of burden of disease and injury attributable to 67 risk factors and risk factor clusters in 21 regions, 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet 2012 Dec 15;380(9859):2224-2260 [FREE Full text] [CrossRef] [Medline]
    15. Goff DC, Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Gibbons R, American College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2014 Jun 24;129(25 Suppl 2):S49-S73 [FREE Full text] [CrossRef] [Medline]
    16. Jackson R, Lawes CMM, Bennett DA, Milne RJ, Rodgers A. Treatment with drugs to lower blood pressure and blood cholesterol based on an individual's absolute cardiovascular risk. Lancet 2005;365(9457):434-441. [CrossRef] [Medline]
    17. Trevena L, Nelson M. Case for action proposal: the right care for the right person at the right time: improving the identification and management of absolute cardiovascular risk in the community. 2014.   URL: https://pdfs.semanticscholar.org/6c27/deb96c3f409d350fff6b93576ca23b41fb93.pdf [accessed 2017-07-31] [WebCite Cache]
    18. Sundström J, Arima H, Woodward M, Jackson R, Karmali K, Lloyd-Jones D, et al. Blood pressure-lowering treatment based on cardiovascular risk: a meta-analysis of individual patient data. Lancet 2014 Aug 16;384(9943):591-598. [CrossRef] [Medline]
    19. Ferket BS, Colkesen EB, Visser JJ, Spronk S, Kraaijenhagen RA, Steyerberg EW, et al. Systematic review of guidelines on cardiovascular risk assessment: which recommendations should clinicians follow for a cardiovascular health check? Arch Intern Med 2010 Jan 11;170(1):27-40. [CrossRef] [Medline]
    20. D'Agostino RB, Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation 2008 Feb 12;117(6):743-753 [FREE Full text] [CrossRef] [Medline]
    21. Bonner C, Jansen J, Newell BR, Irwig L, Glasziou P, Doust J, et al. I don't believe it, but I'd better do something about it: patient experiences of online heart age risk calculators. J Med Internet Res 2014 May 05;16(5):e120 [FREE Full text] [CrossRef] [Medline]
    22. Neufingerl N, Cobain MR, Newson RS. Web-based self-assessment health tools: who are the users and what is the impact of missing input information? J Med Internet Res 2014;16(9):e215 [FREE Full text] [CrossRef] [Medline]
    23. Bonner C, Jansen J, Newell BR, Irwig L, Teixeira-Pinto A, Glasziou P, et al. Is the “heart age” concept helpful or harmful compared to absolute cardiovascular disease risk? An experimental study. Med Decis Making 2015 Nov;35(8):967-978. [CrossRef] [Medline]
    24. Lopez-Gonzalez AA, Aguilo A, Frontera M, Bennasar-Veny M, Campos I, Vicente-Herrero T, et al. Effectiveness of the Heart Age tool for improving modifiable cardiovascular risk factors in a Southern European population: a randomized trial. Eur J Prev Cardiol 2015 Mar;22(3):389-396. [CrossRef] [Medline]
    25. Soureti A, Hurling R, Murray P, van Mechelin W, Cobain M. Evaluation of a cardiovascular disease risk assessment tool for the promotion of healthier lifestyles. Eur J Cardiovasc Prev Rehabil 2010 Oct;17(5):519-523. [CrossRef] [Medline]
    26. Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0, updated March 2011. www.cochrane-handbook.org: The Cochrane Collaboration; 2011.
    27. Marie G. Fighting heart disease: assess risk and save lives. 2011.   URL: http://www.calculators.org/health/heart.php [accessed 2017-07-24] [WebCite Cache]
    28. Trevena LJ, Zikmund-Fisher BJ, Edwards A, Gaissmaier W, Galesic M, Han PKJ, et al. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak 2013;13 Suppl 2:S7 [FREE Full text] [CrossRef] [Medline]
    29. Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns 2014 Sep;96(3):395-403. [CrossRef] [Medline]
    30. Morony S, McCaffery KJ, Kirkendall S, Jansen J, Webster AC. Health literacy demand of printed lifestyle patient information materials aimed at people with chronic kidney disease: are materials easy to understand and act on and do they use meaningful visual aids? J Health Comm 2017 Jan 25;22(2):163-170. [CrossRef]
    31. Cajita MI, Rodney T, Xu J, Hladek M, Han H. Quality and health literacy demand of online heart failure information. J Cardiovasc Nurs 2017;32(2):156-164. [CrossRef] [Medline]
    32. Devine T, Broderick J, Harris LM, Wu H, Hilfiker SW. Making quality health websites a national public health priority: toward quality standards. J Med Internet Res 2016 Aug 02;18(8):e211 [FREE Full text] [CrossRef] [Medline]
    33. Vishnevetsky J, Walters CB, Tan KS. Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT). Patient Educ Couns 2017 Sep 06:1. [CrossRef] [Medline]


    Abbreviations

    ACC: American College of Cardiology
    AHA: American Heart Association
    ARIC: Atherosclerosis Risk in Communities
    ASSIGN: Assessing cardiovascular risk using the Scottish Intercollegiate Guidelines Network
    BNF: British National Formulary
    CHD: coronary heart disease
    CVD: cardiovascular disease
    MESA: Multi-Ethnic Study of Atherosclerosis
    PEMAT-P: Patient Education Material Evaluation Tool for Print Materials


    Edited by G Eysenbach; submitted 31.07.17; peer-reviewed by H Sullivan, B Xie, R Grainger; comments to author 14.09.17; revised version received 30.10.17; accepted 30.11.17; published 01.02.18

    ©Carissa Bonner, Michael Anthony Fajardo, Samuel Hui, Renee Stubbs, Lyndal Trevena. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 01.02.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.