Heart Aware is a cardiovascular health risk assessment tool offered by Navigant Consulting, Inc (Navigant Consulting, Inc, Chicago, IL, USA). The data used in this study came from Heart Aware assessments conducted at 127 clinical sites across the United States between January 1, 2006 and June 30, 2010. Research approval for the data was granted by the Institutional Review Board of Texas A&M University. The risk assessment was administered through the Internet. The assessment began by asking respondents, who voluntarily accessed the survey, a series of self-identifying demographic questions about their race, sex, and age. These respondents were recruited to the survey through marketing activities of the hospitals sponsoring the Heart Aware assessment, such as television media advertising campaigns and print media placed throughout both inpatient and outpatient facilities. Respondents were then asked to report their height, weight, whether they used tobacco, level of physical activity, DBP, SBP, TC, HDL-C, and LDL-C. Respondents were then asked to report the last time their health care provider measured their blood pressure, cholesterol, and checked their diabetes status. Finally, respondents were asked a series of questions about their family history, current medications, and current health history, with specific emphasis on CHD symptoms or diagnoses. As the respondents moved through the assessment, 2 unique tools reported data back to the respondents. First, as questions were answered by the respondents, a visual scale indicating the risk for CHD attributable to each risk factor was displayed. Scales for each risk factor were indicated on a color spectrum from green (low risk) to red (critical risk), and were updated as additional information was provided by the respondent. Second, as the respondents moved through different categories of questions, the tool provided the respondents with education about CHD and associated risk factors. This occurred in the form of text boxes on response pages. Definitions of medical terms were also provided to enhance the respondents’ knowledge of CHD and related risk factors.

The clinical sites that elected to offer the Heart Aware assessment determined the level of risk that would prompt the clinical site to extend an invitation to the respondent for a free on-site clinical risk assessment in which their self-reported CHD risk factor values would be measured by a clinician for comparison and validation. Each site set its own criteria for inviting participants to the free clinical assessment; Navigant did not document this protocol, including the method of extending the invitation to the participants. Anecdotal evidence suggests respondents were more likely to be invited for a clinical assessment if their self-reported values indicated 2 or more CHD risk factors.

The variables analyzed as CHD risk factors were collected from the Heart Aware risk factor assessment. These included self-reported and clinically measured values of TC, HDL-C, LDL-C, SBP, DBP, BMI, as well as DM status and tobacco use. These variables were reported in the clinically assessed dataset as ordinal scales based on their frequencies and clinical guideline ranges. This resulted in ordered categories (referred to as “ranges” below) within each variable as follows: TC (<160 mg/dL, 160-199 mg/dL, 200-239 mg/dL, 240-279 mg/dL, >279 mg/dL), HDL-C (<35 mg/dL, 35-44 mg/dL, 45-49 mg/dL, 50-59 mg/dL, and >59 mg/dL), LDL-C (<100 mg/dL, 100-129 mg/dL, 130-159 mg/dL, 160-189 mg/dL, and >189 mg/dL), SBP (<120 mm Hg, 120-129 mm Hg, 130-139 mm Hg, 140-159 mm Hg, 160-199 mm Hg, and >199 mm Hg), and DBP (<80 mm Hg, 80-84 mm Hg, 85-89 mm Hg, 90-99 mm Hg, 100-114 mm Hg, and >114 mm Hg). DM status and tobacco use were coded as yes or no responses.

Respondents who provided a self-reported health assessment and were chosen for and participated in a free clinical assessment of their CHD risk factors were eligible for inclusion in the sample. Those with a prior history of CVD- and CHD-related procedures, such as stroke, acute myocardial infarction, abdominal aortic aneurysm, cardiac arrest, congestive heart failure, angioplasty, catheterization and stent procedures, heart bypass, and carotid procedures, were excluded from the sample to maintain the integrity of the study objective which was to evaluate how community-dwelling adults who had not received a diagnosis of CHD viewed their risk relative to their actual clinically measured risk for CHD.

Depending on the preferred strategy for addressing missing data, 3 samples were available for analysis. To maximize our understanding of the research questions in the study, all 3 samples were made available for analysis. First, analyses were conducted on the original dataset. Given default settings in Stata version 12 (Stata Corp LLP, College Station, TX, USA), this resulted in a listwise deleted sample. Second, analyses were conducted on the original dataset, but with a change in the default settings. Instead of eliminating cases missing any of the variables being analyzed (as was the case with the listwise deleted sample), cases were only removed on a variable-by-variable analysis basis. This resulted in a pairwise deleted sample. Finally, analyses were conducted on the imputed sample. Results of all analyses performed on all 3 samples were then examined.

Self-reported health status data are known to contain substantial amounts of missing data [18]. Participants often choose not to answer certain questions for a variety of reasons, such as lack of knowledge, time constraints in answering the survey, or a desire not to answer certain questions based on individual preferences. Missing data introduces many analytical challenges, especially relating to biased statistical estimators when making inferences from data [19,20]. To address missing data issues, a 3-step process was used to evaluate the missing data. First, data were analyzed for the degree of missing data, such as the number of missing responses across individual variables and individual cases. Second, the pattern of missing data was analyzed using the “mvpatterns” user-written command [21] in Stata version 12. This command allowed the researcher to determine whether the pattern of missing data was missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR) [20]. Third, based on the result of the first 2 steps noted previously, a methodology was devised to adjust the dataset where appropriate to account for missing data where it was deemed a concern. Several techniques were evaluated for this purpose such as listwise deletion and multiple imputation. Multiple imputation was selected as the preferred method for addressing self-reported missing data because of the large number of missing responses and MAR-identified pattern of missing data discussed later in this paper. Multivariate normal imputation with 5 imputations was used to impute missing values of SBP, DBP, and TC because these variables were most closely aligned with the Framingham 10-year CHD risk model. TC was analyzed because it had more complete self-reported data than HDL-C and LDL-C. These latter variables were key considerations when analyzing missing values in the clinically measured dataset discussed subsequently. Multivariate imputation was chosen because of its ability to take advantage of all variables in the analysis to impute the selected missing variables [22]. Five imputations were selected to balance statistical rigor with processing speed. Independent variables used to impute were selected based on the completeness of data and their theoretical association with the imputed variables. The independent variables used for imputation were age, rural/urban designation, sex, DM status, and BMI range.

The analysis of missing data for the self-reported data was vital to the integrity of the study. Because the self-reported values were used as the basis for choosing individuals for a free clinical assessment, missing data could have profound implications for how participants were selected, possibly resulting in selection bias. Because the clinical sites set their own selection criteria that were not chronicled by Navigant, it was even more important to examine whether the missing self-reported data influenced selection by the clinical sites. This test for selection bias in the clinical assessment process was conducted using the original dataset and the imputed dataset. Using both datasets individually, the analysis was done by examining differences in risk factor variable means between those individuals selected for clinical assessment and those individuals not chosen for clinical assessment. For this purpose, t tests with statistical significance determined at the alpha=.05 level were used.

Although the statistical challenges of missing data noted for the self-reported data also applied to the clinically measured data, there were additional complexities that necessitated a separate analysis of the clinically measured dataset. First, when analyzing clinically measured data, many of the variables used in the Framingham 10-year CHD risk model were reported as ranges by the clinical sites. These ranges corresponded to the ranges used in the Framingham 10-year CHD risk model. Second, the pattern of missing data in the clinically measured dataset was different than the self-reported dataset. As such, methodologies used to address missing data concerns in the clinically measured dataset had to recognize both unique patterns of missing data and the fact the data were now reflected in ranges unlike individual values found in the self-reported dataset. Given the MAR-identified pattern of missing data discussed later in this paper, missing data in the clinically measured dataset was imputed with an ordered logistic model versus the multivariate normal model discussed previously. The ordered logistic imputation was carried out with 5 imputations and was used to impute missing values for ranges of SBP, DBP, TC, HDL-C, and LDL-C because these variables were most closely aligned with the Framingham 10-year CHD risk model. Note the imputation of the clinically measured dataset imputed HDL-C and LDL-C in addition to TC. This was because these variables provided a more detailed view of dyslipidemia relative to TC. It was possible to do these imputations in the clinically measured dataset because there was not as much missing data relative to the self-reported dataset. Independent variables used to impute were selected based on the completeness of data and their theoretical association with the imputed variables. The independent variables used for imputation were age, rural/urban designation, sex, DM status, and BMI range.

Descriptive statistics were conducted on each of the 3 samples. The following variables were dichotomized by sex: age, race/ethnicity, rural/urban designation, SBP, DBP, TC, HDL-C, LDL-C, DM status, and tobacco use. Continuous variables were measured with t statistics, and categorical variables were measured with chi-square statistics. Descriptive analyses were performed using Stata version 12. Test statistics were measured for statistical significance at the alpha=.05 level of statistical significance.

Weighted kappa coefficients of interrater agreement between self-reported and clinically measured CHD risk factors were calculated for males and females. A weighted kappa coefficient was used instead of an unweighted kappa coefficient to account for the degree of discordance between each self-reported and clinically measured observation based on the fact the data were categorized as ordinal ranges. The weighting procedure was performed in Stata version 12 using the “wgt(w)” extension of the “kapci” command. The weighted kappa procedure included 100 repetitions. Each coefficient was reported with its standard error and bias-corrected 95% confidence interval. According to established literature [23], the strength of interrater agreement of the kappa coefficient is described as poor (κ<.00), slight (κ=.00-.20), fair (κ=.21-.40), moderate (κ=.41-.60), substantial (κ=.61-.80), and almost perfect (κ=.81-1.00).

The 10-year CHD risk model from the Framingham Heart Study [17] was used to calculate each respondent’s 10-year CHD risk based on their clinically measured risk factor values. The weighted kappa coefficients were then calculated for each stratum of 10-year CHD risk scores to allow for an evaluation of risk factor agreement by level of CHD risk.

The Heart Aware cardiovascular health risk assessment was taken by 373,085 individuals. The clinical sites provided a clinical assessment to 22,346 (5.99%) of these individuals (note the number of individuals offered an assessment but not taking an assessment was not recorded). Among those responding to the self-reported assessment, 238,081 (63.81%) of the respondents did not answer at least 1 of the risk factor variable questions. Based on the robust information presented in the tables subsequently, extended analysis of the results is provided in the discussion section in this paper. Table 1 reports the results of the nonresponse rate for each risk factor variable in the self-reported dataset and in the imputed self-reported dataset.

Table 2 reports the results of the test of difference in means for the risk factor variables of those selected for clinical assessment from the self-reported dataset. It should be noted the table compares the clinically measured dataset for each variable unadjusted by imputation methods (ie, each self-reported variable has its own sample size). The sample size and very limited amount of missing data for each variable can be examined in Table 3. Table 3 also reports the same type of information as Table 2, but for those selected for clinical assessment adjusted by imputation methods. It is clear from both these tables that those with clinically measured risk factors had significantly higher values (ie, poorer values) than the overall self-reported population. This confirms the recruitment strategy of the clinics to find those at high risk of CHD and invite them for a free clinical screening of their risk factors.

Clinically measured risk factor values were reported for 22,346 individuals. This dataset consisted of clinically measured data that were expressed as ordinal (ie, ranges of clinical values) and binary data (eg, DM status and tobacco use). Among these individuals, 6423 (28.74%) of the respondents did not answer at least 1 of the risk factor variable questions.

Table 3 reports the results of the nonresponse rate for each risk factor variable in the clinically measured dataset, including the self-reported nonresponse rate for the same respondents. Table 3 also reports the results of the nonresponse rate for each risk factor variable in the imputed clinically measured dataset.

Table 4 reports the differences in means and proportions of the CHD risk factor variables by sex within the listwise deleted sample (n=5951).

Table 5 reports the differences in means and proportions of the risk factor variables by sex within the pairwise deleted sample (note the sample size varies by each risk factor variable as indicated in the table).

Table 6 reports the differences in means and proportions of the risk factor variables by sex within the imputed sample (n=16,879).

Upon examining the differences of interrater agreement coefficients by the approach used to address missing data, 2 things become apparent. First, ranges of SBP, DBP, and both tobacco use and DM status were not substantially different based on the approach employed to account for missing data. This was mostly because of fewer instances of missing data than other variables in the original dataset. As such, it is appropriate to use any of the 3 samples to establish findings about interrater agreement relative to these variables in the study. However, given the significant amount of missing data for ranges of TC, HDL-C, and LDL-C, the multiple imputation strategy resulted in more conservative results of interrater agreement than the listwise and pairwise deleted samples. As such, the researcher is cautioned to use these figures when establishing findings from the study. Because of these facts, the multiple imputation sample was deemed the most appropriate for discussing findings of this study. This is because the imputation sample was conservative on the variables with greatest instances of missing data, but consistent with the other 2 methodologies for the variables with fewer instances of missing data.

Figure 1 illustrates the comparative 10-year CHD risk score as established by the Framingham Heart Study [24] for the general male population by age group. The results from the listwise deleted and imputed samples, respectively, are also shown for comparison. Figure 2 illustrates the comparative 10-year CHD risk score as established by the Framingham Heart Study [24] for the general female population by age group. The results from the listwise deleted and imputed samples, respectively, are also shown for comparison.

The most significant finding from this study is the fact a community-based health risk assessment for heart disease that is delivered via the Internet (Heart Aware) yields a sample with markedly lower risk of CHD than suggested by population health data. For example, Figure 1 demonstrates that despite the method used to account for missing data, those males participating in the Heart Aware assessment had, on average, a 10-year CHD risk that was up to 19 percentage points lower than their counterparts of the same age. Likewise, Figure 2 demonstrates that despite the method used to account for missing data, those females participating in the Health Aware assessment had, on average, a 10-year CHD risk that was up to 7 percentage points lower than their counterparts of the same age.

It should be noted this problem is compounded by the fact participants selected for clinical evaluation in this study were hand-picked by the individual hospitals based on their perceived high risk of CHD (ie, a random selection of community-dwelling adults for clinical measurement of CHD risk factors would likely result in samples with lower CHD risk, thereby exacerbating the differences between the risk of samples established by the Heart Aware assessment versus population health data), recognizing that anyone with diagnosed CHD was excluded from the study.

There are several reasons that could explain these discrepancies. The Heart Aware assessments were offered almost entirely via the Internet. This probably resulted in biased selection of participants because those using the Internet are generally more technically savvy, have higher levels of education and income, and are comparatively healthier than non-Internet users [27]. As such, it is not surprising the tool procured a lower-risk population that is not representative of the general population. This raises a very important issue about health risk assessments such as Heart Aware. Obviously, recruitment cost is greatly reduced by using the Internet, especially through a hospital’s existing Internet presence. However, if the data are to be used for public health purposes, how can the data be more representative of the population? One approach could be to expand the methods used to collect the same data, such as using in-clinic kiosks to collect data versus relying on a participant having Internet resources available at home. If financial resources were available, the instrument could be made available through a random-digit dial survey. This method has been shown to improve validity in other CVD-related studies [28]. Finally, a simple and cost-effective method could be to use propensity score matching to create appropriate comparison groups for analysis. This approach is also common in CVD-related studies [29]. However, in the case of the current Heart Aware survey, additional variables need to be collected to account for the underlying demographic differences that are likely associated with the Internet selection bias discussed previously (eg, income, education level, and current health care utilization).

It is difficult to understand the influence of the selection issue on the study results. Although we know the samples in this study were of individuals of much lower CHD risk relative to population health data, and we did exclude those who had already been diagnosed with CHD, the relationship between levels of CHD risk and knowledge of CHD risk factors has yet to be firmly established. In fact, in this study, differences in the interrater agreement of self-reported and clinically measured CHD risk factors varied by sex, individual risk factor, and overall 10-year CHD risk stratum.

As noted previously, the Red Dress symbol and Go Red for Women campaign have been high profile efforts to highlight the fact more females of all ages die of CHD than any other cause of death [30]. Yet, the results of this study indicate a very similar level of awareness of CHD risk factors among the sexes. It is difficult to reconcile this data with the potential success or failure of these high profile campaigns specific to females. Although there may be no marked difference in the interrater agreement between males and females on each measured CHD risk factor in the imputed sample, perhaps awareness would have been worse without the public campaigns focused on females?

What is clear about the differences in the interrater agreement of self-reported and clinically measured CHD risk factors by sex is neither sex has demonstrated a superior understanding of their CHD risk factors. Both sexes demonstrate relatively low levels of agreement on every CHD risk factor. It should also be noted females were generally “healthier” than their male counterparts in this study (see Table 6). It seems rational such a difference would influence CHD risk factor agreement by sex to a greater degree than witnessed in this study. This is an area for continued exploration because it is central to public health policy.

In addition to the noteworthy findings of this study, there are several limitations. First, the most substantial limitation is a challenge to internal validity of the results based on a substantial amount of selection bias that was likely the result of the recruitment method (ie, offering the survey to any interested party through media advertising). However, as Guba [31] reminds us in a classic work on naturalistic studies, the process of determining validity is not comparable with rationalistic designs such as randomized controlled trials. Naturalistic trials have a wide array of tools to complement the rationalistic approach to establishing comparable levels of study integrity and quality [31]. Among these methods are techniques such as triangulation of results, replication, and comprehensive descriptive statistics to ensure a thorough understanding of the sample [31]. If naturalistic designs are fundamentally characterized as research conducted in natural settings versus structured environments such as laboratories [32], then Heart Aware should qualify as a tool used in a naturalistic setting. Because of this paradigm, the study does accomplish some of the processes desired by naturalistic researchers, such as the exhaustive approach to examining missing data, the use of multiple imputation to ensure replication of the results displayed in the imputed sample, and the 100 repetitions conducted on each weighted kappa analysis of interrater agreement between self-reported and clinically measured CHD risk factors. Nevertheless, future research in this area should incorporate some of the suggestions made previously to counteract the apparent selection bias of solely using the Internet for recruitment.

The second limitation is the amount of missing data. Although this study has attempted to mitigate this point with multiple approaches, none of these efforts can fully account for the bias that exists in statistical estimation as a result of missing responses. At a very basic level, the latent traits of missing responses remain unknown even with the most sophisticated missing data techniques. However, it should be noted repetition and replication (as noted previously) somewhat mitigate these biases.

The third limitation of the study is the difference between sexes for baseline health behaviors and clinical values. Although some of the clinical value differences are because of normal differences based on sex, some of the discrepancies are very large indicating females are probably healthier than their male counterparts. This influences the ability to fully understand results of the study by sex.

The fourth limitation of the study is the lack of research on how respondents acquired information about their self-reported risk factor values. There could be an element of self-education or access to professional resources that play a role in the findings of the study.

Finally, it was not possible to exclude individuals from the study who had undiagnosed CHD. The sample likely contained some of these individuals and could have contributed to selection bias concerns.

The findings from this study have a unique place in the literature based on the large sample size, breadth of heart disease self-reported risk factors collected, and the method of data collection (ie, the Internet). However, this was a cross-sectional study that lacks the internal validity of a stronger design such as a randomized controlled trial. Future efforts in this field would benefit from a prospective randomized study design to ensure some of the self-selection biases and other limitations of this study are appropriately addressed.

This study sought to understand which CHD risk factors were best understood by community-dwelling adults who took an Internet-based CHD risk assessment (ie, Heart Aware). It also sought to examine whether such levels of understanding were associated with varying degrees of 10-year CHD risk for each participant. What the study has shown is although all CHD risk factors had suboptimal levels of interrater agreement between self-reported and clinically measured values, the CHD risk factor with the greatest discordance was HDL-C. This is consistent with the literature noted previously. However, this study provides unique support to this finding by incorporating a thorough review of how interrater agreement coefficients change based on approaches to missing data. Because missing data are a key analytical issue in many surveillance studies [33], the current study provides a robust view that supports the findings of interrater agreement for HDL-C in a variety of methodological settings. Further, these findings were drawn from a very large sample across more than 100 hospitals.

Unlike prior research efforts, this study stratified interrater agreement of self-reported and clinically measured CHD risk factors by 10-year CHD risk as established by the Framingham Heart Study [24]. This allowed the current study to make a very important contribution to the literature, the discovery that interrater agreement for HDL-C deteriorates as 10-year CHD risk increases, whereas interrater agreement for LDL-C improves as 10-year CHD risk increases. This is a powerful finding because it not only supports the literature noted previously regarding the lack of knowledge of HDL-C among community-dwelling adults, but it also shows how the same individuals also view LDL-C. This finding has substantial implications for the health literacy, social and behavioral health, and public health implementation science communities. If the evidence of HDL-C as a protective factor for CHD continues to mature, it will be vital to translate these clinical findings into actionable public health information campaigns in the community.

Several broad themes should be drawn from this study. First, tools such as Heart Aware could be a cost-effective way to collect valuable CHD risk factor data. Researchers should begin to think about leveraging such technology by partnering with private sector firms to improve public health datasets. Such efforts can only improve public health surveillance, which is positive for researchers, policymakers, private sector firms, and community-dwelling adults. However, additional recruitment methodologies should be employed (in addition to the Internet) to reduce selection bias. Second, this research confirms the continuing need to educate community-dwelling adults about the need to understand their CHD risk factors. This is especially true regarding HDL-C and LDL-C. Finally, this research raises questions about how to use stratification of CHD risk factor agreement by 10-year CHD risk as a clinical strategy. Very few differences in interrater agreement for any CHD risk factor by 10-year CHD risk were identified in this study. Clinicians may want to consider additional strategies to improve CHD risk factor knowledge among those who currently exhibit the greatest chance of a CHD event in the next 10 years.