Proficiency in carrying out online health information-seeking behaviors varies by sociodemographic factors, including age [7]. For example, greater adoption of the Internet by older adults has increased the accessibility of health information to this subset of the population [8,9]. One recent study in the United States showed that Web adoption among older adults is climbing, with 67% of people over 65 years of age using the Internet and more than 40% using smartphones [10]. Over 50% of US adults aged 35 to 60 years reported searching for online health information, while only 31% over the age of 60 years reported doing so [11]. Older adults need high-quality, relevant, and accurate health information regarding age-related physical conditions and ailments that require regular and consistent medical attention [12,13]. However, research suggests that most older adults do not access high-quality health information that addresses their health concerns [14].

There are several reasons why older adults may be unable to benefit from increased access to online health information. Older adult populations report high computer anxiety, which compromises their ability to carry out functional tasks using Internet-based technologies [15]. Only 26% of older adult Internet users reported feeling confident when using the Internet to complete daily tasks [10]. This lack of confidence using digital devices often leads to lack of Internet use for health information among older adult populations [10,16-18]. Nevertheless, older adults who overcome anxiety toward using health information technology demonstrate greater patient activation (ie, enhanced knowledge, skills, and confidence a person has in managing their own health and health care) and are more satisfied after talking with their provider about their own medical questions [19,20].

Moreover, routine online health information seeking has the potential to motivate older adults living with chronic disease to become more proactive in their health care decision making [21,22]. Because it is very likely that older adults will increasingly use the Internet to access health information to improve their health, it is important to measure the extent to which they have the capacity to search for, retrieve, and evaluate health-related resources that they come across online (ie, eHealth literacy).

eHealth literacy was originally defined by Norman and Skinner [23] as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem.” To conceptualize eHealth literacy, Norman and Skinner [23] used the metaphor of a lily flower with 6 discrete petals (literacies) feeding into a core pistil. They categorized the core literacies proposed to contribute to eHealth literacy as being either context specific (ie, health, computer, and science literacies) or analytic specific (ie, traditional and numeracy, information, and media literacies). The concept of eHealth literacy is dynamic and evolving, meaning it varies per a variety of individual and contextual factors, including an individual’s health status, their purpose(s) for seeking health information, and the technology they select to access health information. Recent research suggests that people with greater eHealth literacy are more informed health decision makers [24], which ultimately increases their capacity to engage in health protective behaviors [25] and improve their quality of life [26]. While several studies have examined eHealth literacy, rigorous measurement of the 6 constituent eHealth literacies is underdeveloped and presents an ongoing challenge for health promotion researchers.

In 2006, Norman and Skinner [27] developed the eHealth Literacy Scale (eHEALS), an 8-item rating scale that measures consumers’ knowledge of and perceived confidence in their ability to seek, understand, and evaluate health information obtained from the Internet to address health-related concerns. Scores from eHEALS have supported its reliability as a unidimensional scale in diverse populations, including adolescents [27], college students [28], adults in the general US population [28], older adults recruited on the Internet [29], and people living with chronic disease [30,31]. eHEALS has been translated into many different languages and administered in countries around the world (eg, Germany, Italy, the Netherlands, Israel, and China).

Several studies have explored the dimensionality of data produced by eHEALS, reporting varied results. This literature describes some potential problems related to the internal structure of the eHEALS. Specifically, the number of factors (and factor loadings) derived in measurement studies of the eHEALS have shown some variability. Two recent studies reported that up to 3 unique, yet highly correlated, factors may be present when the scale is administered to older adults [30,32]. However, studies reporting the presence of multiple subscales have yet to explicate which eHEALS items load onto distinct factors (or constructs) when eHEALS is completed by older adults [29,30,32]. This variability has caused some difficulty when attempting to define what these unique factors, or subscales, are actually measuring.

Soellner and colleagues [33] translated the eHEALS into German and found that, despite poor global model fit, data from 18-year-old university students may best fit a 2-factor model, where eHEALS items measure online health information seeking (items 1-5, and 8) and online health resource appraisal (items 7 and 8). Neter and colleagues [7] also found adequate global model fit with a 2-factor model of eHEALS data when collected among adults over 21 years of age. This 2-factor structure consisted of 1 factor measuring online health information seeking (items 1-3) and another measuring online health resource appraisal (items 4-8). It should be noted, however, that 1 study [33] primarily consisted of adolescents. Younger people are more likely to report higher eHealth literacy than their older counterparts [7,31]. The factor structure and variance of eHEALS scores may differ as a function of age, which could influence results from eHEALS studies including younger versus older samples. Diviani et al [34] conducted a validation study of the Italian version of eHEALS administered among young to middle-aged adults (mean age 37.37 years, SD 13.78). Confirmatory factor analysis results showed suboptimal model fit among 2 rival models (1-factor structure vs 2-factor structure), yet parametric and nonparametric item response theory (IRT) analyses confirmed that the single-factor model best fit the data in the study sample. However, studies reporting the presence of multiple subscales have yet to explicate which eHEALS items load onto distinct factors (or constructs) when eHEALS is completed by older adults [29,30,32].

It is also important to note that the mode of survey administration can affect the reliability and validity evidence of survey data [35]. Many studies examining the internal structure of eHEALS data collected from older adults have only used Web-based survey methods [29,30,32]. Web-based surveys have several advantages, including time and cost efficiencies, but they are prone to response bias, especially when respondents demonstrate concerns about the privacy of disclosing information through Web-based survey portals [36]. Also, analyzing eHEALS data collected from only active Internet users may reduce the quality of reliability and validity assessments due to sampling bias. Older adults who use the Internet to complete the eHEALS are more likely to be more confident in their online health information-seeking skills; thus, solely relying on Web-based survey methods to establish evidence for the validity of eHEALS scores may introduce measurement bias. Administering Web-based versions of eHEALS to older Internet users may skew data toward respondents with high overall eHealth literacy, which may partly explain why existing studies report moderate to high eHealth literacy in older adult populations.

Dillman [37] recommended use of telephone-based surveys for collecting data among older populations, who often feel more comfortable answering questions asked by an actual person rather than via online or paper-and-pencil questionnaires. In a recent study, Neter and Brainin [38] conducted a nationally representative random digital dial telephone household survey of Israeli adults aged 50 years and older to determine their perceived eHealth literacy as measured by eHEALS. In this older population, perceived eHealth literacy was judged to be moderate (mean 3.17, SD 0.93), with a moderate correlation established between perceived and actual eHealth literacy (r=.34, P=.01). However, no psychometric data on eHEALS responses was reported in this age-restricted (50 years of age and older) sample. Therefore, much variability has been documented in the literature and has led to difficulty defining what the unique factors, or eHEALS subscales, may be measuring. These discrepancies in confirmatory factor analysis and IRT analysis results highlight the importance of conducting additional psychometric research that considers differences in eHEALS item measurement, factor structure, and item difficulty among older adults. The purpose of this study was to examine the reliability and explore the internal structure of eHEALS data, when the scale is administered to older adults using telephone-based survey methods.

We conducted a cross-sectional landline telephone survey as part of the Florida Consumer Confidence Index (F-CCI) Survey [39]. At least 500 households in the US state of Florida were contacted over 1 month. A minimum of 10 call attempts per household were made every Monday through Friday (between 9:00 AM and 9:00 PM), Saturday (between 12:00 PM and 6:00 PM), and Sunday (between 3:00 PM and 9:00 PM) using the random digit dialing method. The Institutional Review Board at the University of Florida approved the conduct of this study. Overall, 6695 calls were placed, and 493 individuals (response rate 7%) agreed to participate in the telephone survey. Participants were not provided incentives as part of participating in the F-CCI. We included data from these individuals in the main analyses if respondents reported being (1) at least 50 years old, and (2) Internet or email users. We selected the age cutoff based on Watkins and Xie’s [40] systematic review of eHealth literacy interventions for older adults, citing that chronological age for the older population “can range from 50 to over 100” years, and the age range of 50 years and older “is consistent with growing appreciation of the role that health behavior interventions play in healthy aging for those under age 65” years (pg e255). While screening participants for this study, we found that 393 F-CCI Survey respondents reported being at least 50 years old, yet 110 responded “no” when asked if they used the Internet or email. Therefore, the final sample size for this study was N=283.

An exploratory structural equation modeling (E-SEM) approach [41] using the weighted least squares and adjusted means and variances (WLSMV) estimator examined the model fit of eHEALS scores with 1-, 2-, and 3-factor structures. This model uses an exploratory factor analysis measurement model and applies a structural equation model to describe (1) which items significantly load onto the extracted factor(s); (2) the dimensionality or number of factors (or subscales) produced; and (3) the relationships between factors (if more than 1 factor is extracted). The following global model fit indices provided evidence of good model fit [42]: (1) root mean square error of approximation (RMSEA) value close to .06; (2) comparative fit index (CFI) value >.95; (3) Tucker-Lewis index (TLI) value >.95; and (4) nonstatistically significant chi-square test. We evaluated factor loadings of each item for statistical significance (P<.05) and computed fit indices for all 3 factor structures to determine the best overall model fit. We used Mplus v7.3 (Muthén & Muthén; [43]) to conduct all E-SEM analyses.

Following E-SEM analyses, we used the partial credit model (PCM), an IRT analysis [44,45], to explore the internal structure of the self-reported polytomous (ie, more than 2 possible response options) eHEALS data. This analysis was appropriate given that the final sample size (N=283) was over 200 cases and greater than 10 times the number of eHEALS items (ie, 8) [46,47]. PCM constrains item discrimination, or the strength (slope) of the relationship between responses and a latent trait. This provides important information on which response options have the greatest probability of being answered at a particular theta (ie, a person’s latent trait score) level on the latent continuum. Information from PCM analyses helps to evaluate stability across items, which reduces the potential for item bias [48,49]. Allowing step variability to vary across items provides useful information about the range of difficulties measured in a scale, including whether differences in step difficulties exist across items. RStudio’s eRm software package version 0.15-7 (R Foundation; [50]) computed all PCM estimates.

Finally, Linacre’s guidelines [46] for optimizing rating scales under IRT assumptions informed item fit analyses that calculated step difficulties of each response option. Optimized rating scales have threshold values (ie, relative difficulties to advance from one response option to the other) that increase across the theta continuum, which helps confirm that higher response options coincide with greater ability levels. Relative difficulties across response options helped to determine how precisely each eHEALS item was measured on the latent continuum. Values for each item that advanced less than 1.4 logits indicated a lack of variability across response categories, whereas values advancing more than 5.0 logits indicated extremely high variability, or low precision, between response categories.

Infit and outfit mean square (MSQ) and t statistics determined the level of noise or randomness in item response options. For outfit MSQ values, any value greater than 1.5 indicates unpredictable random error, whereas a value less than 1.0 indicates a degree of overpredictability and nonrandom error. Values less than 0.5 are interpreted as troublesome for overfit. For outfit t statistics, a value greater than 2.0 indicates underfit and less than –2.0 indicates overfit [50]. Measurement stability, which describes adequate item placement across the latent continuum, is determined based on adequate item reliability (>.80) and satisfactory item separability (>2.0) [51].

As reported by Tennant and colleagues [14], the mean age of respondents was 67.46 years (SD 9.98 years). Most respondents were white (n=252, 89.1%) and non-Hispanic (n=264, 93.3%). A little over half identified as being male (n=155, 54.8%). Over three-quarters of the sample (n=215, 75.9%) reported at least some college-level education, and over half (n=138, 60.4%) reported earning more than US $50,000 per year. Additionally, nearly three-quarters of respondents reported their health as being “good” (n=72, 25.1%), “very good” (n=103, 36.4%), or “excellent” (n=62, 21.9%). A little more than one-third of respondents reported accessing social media (n=101, 35.7%) to locate or share health information.

Total eHEALS scores ranged from 11 to 40 (mean 29.05, SD 5.75). Table 1 presents the mean (SD) score for the response to each item. Internal consistency estimates of eHEALS data collected in this study were relatively high (Cronbach alpha=.91).

Table 2 lists global model fit statistics and factor loadings for models fitting 1, 2, and 3 factors.

The item reliability of eHEALS scores in this sample was estimated at .92 (observed variance=4.58), while the item separation index was 11.31. Both values were indicative of high reliability and stability across the latent continuum.

Table 4 shows that Linacre’s assumption of monotonicity was satisfied, with thresholds (ie, relative difficulty advancing from response options) increasing across the theta continuum, as demonstrated in the item characteristic curves and reported threshold values, confirming that greater eHealth literacy coincided with higher response options. However, not all step difficulties advanced from 1.4 to 5 logits. Relative difficulty moving from “strongly disagree” to “disagree” for almost all items was less than 1.4 logits, except for eHEALS item 8 (“I feel confident in using information from the Internet to make health decisions”), where it was 1.96. Relative difficulty moving from “agree” to “strongly agree” was within the acceptable range for all items (ie, all below 5.0 logits), but they were quite large as compared with advances in the relative difficulty for thresholds 1 (“strongly disagree” to “disagree”), 2 (“disagree” to “neutral”), and 3 (“neutral” to “agree”) [46].

Table 5 shows that all outfit MSQ values were <2.0 yet closer to 1.0, which suggested an optimal degree of randomness in responses to eHEALS items. However, the outfit MSQ values for items 2 to 5 fell well below 1.0, suggesting some level of overpredictability (ie, respondents with a particular eHealth literacy level were responding to items 2 to 5 using similar response options). Subsequently, we noted that infit t statistics for items 2 to 5 were all below –2.0, which is outside of the acceptable range of –2 to 2.

Figure 1 depicts item and test information functions. The test information curve shows a high degree of information with minimal standard measurement error around theta levels –2 to 2 on the latent continuum. The test information curve shows that eHEALS provides some degree of information for participants at the higher end of the latent continuum, but reliability and validity evidence for this level of information is likely unstable. Moreover, the test information function is positively skewed, rather than bell shaped. This result indicates that eHEALS items may capture more information about eHealth literacy among participants who place higher on the latent continuum (ie, those with high eHealth literacy) than among those at the lower end of the continuum (ie, those with low eHealth literacy).

Item information curves showed that all eHEALS items followed a similar curvature pattern, yet the peak of most item curves (greatest amount of information) were plotted at different points on the latent continuum. Despite test information functions that were positively skewed, item information curves suggested that each eHEALS item made important contributions to the complete measure of eHealth literacy. Interestingly, information obtained from items 2 to 5 did not vary across different points on the latent continuum. Therefore, items 2 to 5 may produce a similar amount of information at each point on the latent continuum.

There are several limitations to note in this study. First, this was a cross-sectional study and, therefore, we were unable to compute test-retest reliability or predictive validity estimates. Second, our analyses used telephone survey data with a very low overall response rate (7.4%), resulting in the possibility of nonresponse bias. Third, comparative measures of model fit did not inform decisions regarding the optimal internal structure of eHEALS data collected in this study. Comparative fit measures such as the Akaike information criterion can only be estimated with maximum likelihood data extraction methods, which Mplus v7.3 does not allow for under the WLSMV estimator. We selected the WLSMV estimator to examine model fit in this study for several reasons: (1) WLSMV estimation compensates more effectively than the maximum likelihood estimation for bias due to ordinal response options in the eHEALS, and (2) WLSMV estimation is less likely to produce unrealistic indices of overall model fit [55,56]. Therefore, we based conclusions regarding the internal structure of eHEALS data on noncriterion-based judgments made through interpretation of E-SEM and IRT analyses results.

Fourth, this study contacted participants through a landline sampling technique, which may have selectively excluded individuals who may only own a mobile phone. Although this telephone sampling method targeted older adults living in the state of Florida, this state is home to the greatest proportion (19.1%) of older adults in the United States [57].

Fifth, this study examined eHEALS responses derived from telephone administration of the survey, despite all participants reporting use of the Internet or email. Widespread adoption of the Internet and mobile phone technology has contributed to nonuse of landline telephones. We did not account for mobile devices and cellular telephones, which are increasingly being used by middle- to older-aged adults [58], in this sample. Partnering with community-engaged research programs or local community organizations to reach older adults via telephone may enhance recruitment efforts in this population traditionally underrepresented in health-related survey research.

Assessing consumer comfort and self-efficacy in using technology to access online health resources can help identify skill gaps and gauge the likelihood that users will be successful when using the Internet to access relevant health information [23]. Results from this study suggest that administering eHEALS to older adults via telephone produces a reliable measure with scores that possess sufficient construct validity evidence. Specifically, results from this study support the previously reported unidimensionality of eHEALS scores. Among older adults, however, there is potential for additional underlying subscales to measure older adults’ confidence to locate, use, and evaluate online health information. As older Internet users continue to visit online support groups and discussion forums to find new information about health care perspectives and experiences, it will be important to consider modifying the original eHEALS to adequately measure online health information-seeking behaviors in older populations.