This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Online health information is particularly important for cardiovascular disease (CVD) prevention, where lifestyle changes are recommended until risk becomes high enough to warrant pharmacological intervention. Online information is abundant, but the quality is often poor and many people do not have adequate health literacy to access, understand, and use it effectively.
This project aimed to review and evaluate the suitability of online CVD risk calculators for use by low health literate consumers in terms of clinical validity, understandability, and actionability.
This systematic review of public websites from August to November 2016 used evaluation of clinical validity based on a high-risk patient profile and assessment of understandability and actionability using Patient Education Material Evaluation Tool for Print Materials.
A total of 67 unique webpages and 73 unique CVD risk calculators were identified. The same high-risk patient profile produced widely variable CVD risk estimates, ranging from as little as 3% to as high as a 43% risk of a CVD event over the next 10 years. One-quarter (25%) of risk calculators did not specify what model these estimates were based on. The most common clinical model was Framingham (44%), and most calculators (77%) provided a 10-year CVD risk estimate. The calculators scored moderately on understandability (mean score 64%) and poorly on actionability (mean score 19%). The absolute percentage risk was stated in most (but not all) calculators (79%), and only 18% included graphical formats consistent with recommended risk communication guidelines.
There is a plethora of online CVD risk calculators available, but they are not readily understandable and their actionability is poor. Entering the same clinical information produces widely varying results with little explanation. Developers need to address actionability as well as clinical validity and understandability to improve usefulness to consumers with low health literacy.
Online health information may be the first step toward seeking professional medical advice, so the quality of this information is important: is it clinically valid, does it communicate risk effectively, is it understandable to the user, and what actions does it prompt? Unfortunately, the majority of users may not have the necessary skills to effectively evaluate these issues. Health literacy is the ability to access, understand, and make use of health information and services [
CVD is the leading cause of mortality and morbidity worldwide, but its incidence can be reduced through risk factor modification via lifestyle change and/or medication [
Previous research indicates that online CVD risk calculators can be easily misunderstood. Users may enter their risk factors incorrectly, the provision of multiple risk formats can be confusing if not explicitly explained, and the risk calculators themselves may make assumptions about missing data that lead to less accurate results [
This study aimed to systematically review publicly available online risk calculators for CVD and evaluate them on criteria relevant to health literacy (clinical validity, risk communication, understandability, and actionability).
The general approach for this study was to follow a systematic review process using 2 independent searchers (SH, RS) who qualitatively described each risk calculator and evaluated them quantitatively based on the validated Patient Education Material Evaluation Tool for Print Materials (PEMAT-P) scale. For the search and evaluation, we used a third rater (MF) to resolve discrepancies and reach consensus in accordance with section 7.6 of the Cochrane Handbook for Systematic Review of Interventions [
Since there were no participants in this study and the data was based on publicly available websites, an ethics application was not required.
Online risk calculators were considered if they met the following inclusion criteria: (1) assessed risk of future CVD in individuals without a previous CVD event, (2) were available without the need for registration or payment, and (3) were interactive. Calculators were not considered if they were downloadable files such as an Excel (Microsoft Corp) spreadsheet or PDF, addressed absolute risk of future cardiovascular events in people with atrial fibrillation, or did not provide a risk result for the end user.
There were 2 main search strategies for identifying Web addresses that contained the CVD risk calculators. The first strategy was to access predetermined reputable websites including 6 national heart foundation websites (Australian National Vascular Disease Prevention Alliance, the National Heart Foundation of New Zealand, the Joint British Societies, the UK National Health Service, the American Heart Association, and the American College of Cardiology) and a not-for-profit source [
The searchers (SH, RS) conducted this search as part of a Master of Public Health degree capstone unit from August to November 2016. In March 2017, an independent member of the team (MF) reconciled these search results based on the record of screened Web addresses provided by the original searchers (see
Search strategy and results (updated with higher res image).
The 2 searchers (SH, RS) developed the framework for basic data extraction by qualitatively describing the content of the risk calculator results for the high-risk profile. From this, a standard form was developed (CB, MF) to numerically record basic descriptive data for each calculator: risk model used, how the risk was presented (eg, relative risk, absolute risk, life expectancy, graphical formats) and recommended actions for the high-risk profile (eg, take medication, change lifestyle, see general practitioner). The third rater (MF) then extracted the data numerically, with any uncertainties discussed with the lead researcher (CB) to reach consensus.
The 2 searchers (SH, RS) also rated the content of each risk calculator using a validated tool, the PEMAT-P [
A predefined high-risk cardiovascular profile for a hypothetical patient was used to assess the clinical validity of each calculator. This was a 65-year-old male smoker with systolic/diastolic blood pressure of 130/80 mm Hg, total/high-density lipoprotein cholesterol ratio of 6, and body mass index of 26 kg/m2. Where risk calculators had additional factors, the question was left blank (if possible) or an answer was given that either indicated the middle of the range or provided no additional risk on top of the risk profile (eg, a low-risk ethnicity, no history of CVD or taking medication). The 2 independent raters (SH, RS) created descriptive lists of the different clinical risk models, risk formats, and recommended actions for each calculator based on the high-risk profile. The third rater (MF) then used this framework to code the full dataset after discussion with CB, using the same high-risk profile.
This search yielded 67 unique webpages (see
The descriptive characteristics of the calculators are provided in
The calculators used a variety of published risk models but the most common were those used in clinical practice guidelines: Framingham (44%), American College of Cardiology/American Heart Association (ACC/AHA) (10%), QRISK2 (5%), Reynolds (4%), and Assessing cardiovascular risk using the Scottish Intercollegiate Guidelines Network (ASSIGN) (3%). One-quarter did not specify the underlying model (25%). The outcomes included CVD/coronary heart disease (CHD), angina, heart attack/myocardial infarction, stroke, heart failure, kidney disease, diabetes, and CVD/CHD death. Most calculators used a 10-year time frame (77%) but some used 5-year (7%), 30-year (3%), or lifetime (7%) risk, and one allowed different estimates for the range of 1 to 10 years (1%). Many risk calculators did not state specific outcomes beyond mentioning CVD risk, and some did not state the time frame.
The absolute percentage risk was stated in most but not all calculators (79%). Other risk formats included categorical risk with 2 to 4 groups ranging from low to high (32%), a verbal description of the frequency such as 8 in 100 (18%), heart age (10%), life expectancy (3%), and relative risk (3%). In 34 risk calculators (47%) only 1 risk format was presented, and in 39 (53%) risk calculators 2 to 5 risk formats were presented. The risk profile outlined previously yielded highly variable results depending on the model and outcomes used, with an absolute risk ranging from 3% to 43% over 10 years, a heart age of 68 to 86 years, a life expectancy of 79 to 84 years, and a relative risk of 1.8 to 2.1 compared to a healthy person’s risk. Visual aids included icon arrays or pictographs (18%), bar or line graphs (16%), and charts showing risk level (10%).
For the PEMAT-P evaluation, the calculators scored moderately on understandability and poorly on actionability. The average understandability score was 64% (SD 20%) which ranged from 30% to 100%, and the average actionability score was 19% (SD 26%) which ranged from 0 to 100%. Screenshots from very high-scoring examples for understandability (ID14) and actionability (ID66) are shown in
Characteristics of final calculators (n=73).
Characteristic | Count, n (%) | |
Framingham | 32 (44) | |
Not stated | 18 (25) | |
ACC/AHAa | 7 (10) | |
QRISK2 | 4 (5) | |
Reynolds Risk Score | 3 (4) | |
ASSIGNb | 2 (3) | |
ARICc Study | 1 (1) | |
BNFd | 1 (1) | |
Health Professional Follow-Up Study and Nurses’ Health Study | 1 (1) | |
MESAe | 1 (1) | |
Pocock et al (2001) | 1 (1) | |
QStroke | 1 (1) | |
Strong Heart Study | 1 (1) | |
Absolute risk | 58 (79) | |
Categorical risk | 23 (32) | |
Frequency | 13 (18) | |
Heart age | 7 (10) | |
Life expectancy | 2 (3) | |
Relative risk | 2 (3) | |
Icon array/pictograph | 13 (18) | |
Graphs | 12 (16) | |
Charts | 7 (10) | |
Stop smoking | 21 (29) | |
Lower cholesterol/take cholesterol medication | 21 (29) | |
Lower blood pressure/take blood pressure medication | 14 (19) | |
Improve diet | 10 (14) | |
Increase physical activity | 9 (12) | |
Seek doctor’s advice | 9 (12) | |
Take aspirin | 6 (8) | |
Address body mass index | 3 (4) |
aACC/AHA: American College of Cardiology/American Heart Association.
bASSIGN: Assessing cardiovascular risk using the Scottish Intercollegiate Guidelines Network.
cARIC: Atherosclerosis Risk in Communities.
dBNF: British National Formulary.
eMESA: Multi-Ethnic Study of Atherosclerosis.
An example of a risk calculator with a high understandability Patient Education Material Evaluation Tool for Print Materials score.
An example of a risk calculator with a high actionability Patient Education Material Evaluation Tool for Print Materials score.
This review found 73 unique CVD risk calculators available to the public online that use a wide variety of risk models, risk communication formats, understandability, and actionability. Of particular concern is the variation in CVD risk estimates based on entering the same hypothetical high-risk patient data with little explanation for why this would occur: ranging from as little as 3% to as high as a 43% risk of a CVD event over the next 10 years. One-quarter (25%) of the risk calculators did not specify the underlying model, study, calculations, or assumptions, so it not possible to assess their validity or the reasons for variation. The remaining three-quarters (75%) did specify enough information to determine the underlying algorithm, where differences in the study population (eg, US-based Framingham versus UK-based QRISK), included risk factors (eg, whether less directly predictive factors like body mass index were included as well as strong clinical indicators like cholesterol), and CVD outcomes (eg, mortality only, nonfatal heart attack and stroke, angina) explain the discrepant risk results for the same profile. In the academic literature, there are over 17 validated CVD risk models [
In terms of risk communication, absolute risk should ideally be presented in a variety of formats to cater to different needs and learning styles [
The PEMAT-P evaluation was chosen for its unique focus on reducing health literacy demand through both understandability and actionability [
The findings of this study are comparable to broader literature on the quality of online health information; a review showing 70% of studies evaluating 5941 websites concluded that higher quality is needed [
The strengths of this study include a systematic review and evaluation process with multiple independent searchers/raters. The main limitation is the replicability of conducting a systematic search using online search engines like Google. The dynamic nature of the Web with constant variation in website content and metadata means that no search is perfectly replicable even though the cache was cleared between search terms. However, the methods used are likely to have captured the most common and popular search results, since many duplicates were removed between the 2 searchers. It is likely that additional calculators existed at the time of the search and could potentially have been found by a different searcher, search engine, or geographical location, but this study provides a comprehensive list of accessible calculators at the time of searching.
The variable reliability of the PEMAT-P items was slightly lower than previous research [
The plethora of calculators and wide variation in results from the same input have the potential to confuse and harm the general public if appropriate medical advice is not sought. Actionability scores are poor on average, and minimum risk communication standards are not being met. Future research evaluating the suitability of online risk calculators for low health literacy users would benefit from a revised version of the PEMAT-P designed specifically for interactive online formats. Existing items relevant to health literacy may need to be clarified, including (1) better definition for the presence/absence of a table, since website content is often built around a table format, (2) instructions for dealing with interactive graphs based on buttons, (3) a definition of short material for webpages, (4) how to address subtle visual aids such as color, size, or positioning which are more heterogeneous in interactive online formats, and (5) how to best address actionability items with general health advice versus personalized advice based on the risk result. In addition, concepts from eHealth evaluation tools could be incorporated, including (1) ease of navigation through the calculator; (2) presence/absence of distractions in the webpage (eg, advertisements, pop-ups); and (3) ability to save, print, or email the results and recommended actions. Finally, additional information is needed to enable clinicians and consumers to determine whether health risk calculators give a reliable estimate, including the model or algorithm used (eg, study name/reference, outcomes measured, and time frame), what population the calculation has been validated in (eg, age, sex, and ethnicity), an explanation of how this relates to current clinical guidelines, and when the evidence for the calculation was last updated.
Online CVD risk calculators produce highly variable results for the same person with little explanation for why this would occur. Differences in the models used, risk factors included, risk communication formats presented, and actions specified explain part of this variation, but one-quarter of risk calculators did not specify any underlying assumptions. Health professionals should be aware of the reasons for conflicting results that patients might encounter, and developers need to address actionability as well as clinical validity and understandability to improve usefulness to the majority of the population with low health literacy. Country-specific calculators that match national clinical guidelines and build on examples with high understandability and actionability scores would benefit both health professionals and consumers.
Risk calculator archive and Patient Education Material Evaluation Tool for Print Materials scores.
American College of Cardiology
American Heart Association
Atherosclerosis Risk in Communities
Assessing cardiovascular risk using the Scottish Intercollegiate Guidelines Network
British National Formulary
coronary heart disease
cardiovascular disease
Multi-Ethnic Study of Atherosclerosis
Patient Education Material Evaluation Tool for Print Materials
The study was supported by the National Health and Medical Research Council of Australia through the Ask, Share, Know: Rapid Evidence for General Practice Decisions Centre for Research Excellence. CB contributed to the conceptualization, methodology, data analysis and interpretation, and drafting the manuscript. MF contributed to the methodology, data analysis, and revising the manuscript. SH contributed to the methodology, data collection, and revising the manuscript. RS contributed to the methodology, data collection, and revising the manuscript. LT contributed to the conceptualization, methodology, and revising the manuscript.
None declared.