The Learning Network of Chartered Value Exchanges (CVEs) is a program that has supported transparency since 2004 and is sponsored by the Agency for Healthcare Research and Quality (AHRQ). The CVEs are multistakeholder state or local quality collaboratives that are investing significant resources in online public reports of hospital and physician quality in their communities. The CVEs involve more than 550 health care leaders and represent more than 124 million lives, more than one-third of the US population [14]. Because of the CVEs’ long experience and broad catchment area, we used the network of CVEs for this study.

All 24 CVE’s were invited to participate, with 22 CVE or CVE-affiliated websites active. Of these, 18 websites participated. The websites were public quality reports of hospitals, outpatient providers (medical groups or clinics), or both. Websites reported on providers within a state, a region in a state, or a county. Quality measures for hospitals were commonly Center for Medicaid and Medicare Services (CMS) measures, although some websites also included measures of maternity and neonatal care. Quality measures for outpatient providers were often Healthcare Effectiveness Data and Information Set (HEDIS) measures and Consumer Assessment of Healthcare Providers and Systems (CAHPS) patient experience measures. Narrative patient comments were not shown on any participating website.

Further descriptions of CVE groups and websites are available [14]. Participating websites agreed to participate on the condition that we not identify the websites and their performance individually. Multimedia Appendix 1 shows a screenshot from one of the websites at the time of the study, whose sponsors gave permission to share it.

We gathered Web analytics data from February to August 2011 using Google Analytics [15]. We excluded data from the IP addresses of computers of the report sponsor organization and of any external vendors hosting the online reports.

Collected variables were number of unique visitors and percent of unique visitors arriving via 3 methods: search engine queries (“search traffic”), clicking on a link from a different website (“referral traffic”), or directly entering a URL into a Web browser or clicking on a link in an email, word processing document, or document in Portable Document Format (PDF) (“direct traffic”).

The population density varied among the catchment areas the websites served. To generate population-adjusted website traffic from the absolute number of unique visitors, we used Census Bureau catchment area counts of 100,000 Internet-using households as the denominator under total number of unique visitors to calculate “per capita traffic” to each website [16,17].

We also collected search terms (eg, “best doctors San Francisco”) for all search engine traffic. The primary author (NSB) and a research staff member (RAP) organized the search terms into categories using an iterative process: both investigators read the 50 most common search terms for each website then discussed the various categories of search terms to create a codebook for categories. RAP then coded the top 50 search terms for each website and NSB reviewed the initial coding and discussed any code changes. They then combined codes into larger groupings for the final analysis. The codebook and documentation of the coding process is available on request.

We could not link the Web analytics data to survey responses because the analytics data do not include visitor IP addresses and because they are reported in aggregate.

The primary aim of the survey was to provide information on report visitors’ use and perceptions of the value of the public reporting websites. See Multimedia Appendix 2 for the full Checklist for Reporting Results of Internet E-Surveys (CHERRIES) checklist for reporting of Internet e-surveys and for additional description of survey development [18]. We drafted the AHRQ Public Report [19] surveys based on the authors’ expert knowledge about online public reports and drawing from existing surveys that participating public reporting websites were using.

Survey items fell into the following categories: demographics, purpose of the visit, medical topics of interest, and areas of quality measurement of interest. Visitors were shown questions within each topic tailored to their persona type (eg, patient, friend or family member, health care professional, insurer, employer or labor union, researcher, media, lawyer, legislator). We focused on the results for the consumers (patients or friends or family members) and health care professionals in this paper because these were the largest groups of respondents and are the ones most likely to use the reports for choice or for performance improvement. The surveys are available in Multimedia Appendix 2.

We surveyed participating website visitors from February to August 2011. We used an “open survey” design in which all visitors viewing at least 1 page with access to quality performance measures were offered the opportunity to take the survey. The invitation appeared in a pop-up window with directions to take the survey at the end of the session. See Multimedia Appendix 1 for images of invitation and survey.

We framed the survey respondents as a group interested enough in the website content to spend time on an online survey afterward. The implication is that although there might be a low response rate, the responses we received would be from people who are more likely to be potentially influenced by the report. We adopted this frame because low response rates are a known limitation for website surveys because a proportion of website visitors are searching for other content or have limited attention or time for a survey while online. For instance, Kaiser researchers had a 17% response rate in an online survey of users of a secure online personal health record who were presumably more engaged than one-time visitors to a website [20]. One approach to the known low response rate is to invite only visitors who interact extensively with a website to answer a survey, thus creating a smaller response rate denominator. We chose to invite all visitors because some websites have quality information on only 1 or 2 pages.

To decrease response burden, we programmed the survey software to show each consumer (patient or friend/family member) a randomized set of 3 of 5 item groupings (eg, purpose of visit, demographics, topics of interest), leading to smaller consumer sample sizes for each set of questions than if all consumers had answered all items. As noted in our results, the denominators for each of these item sets only included those who were randomized to see those questions.

We coded free-text survey responses using the existing survey item options and categorizing responses that did not fit into an existing option as “other.” We allowed a new category to be formed if it occurred more than 10 times and put the answers for an existing option into “other” if the option was chosen less than 10 times. The only option affected was the consumer primary purpose of visit, for which “learn about a disease” was included in “other.”

For the 18 websites, there was an average of 1440 unique visitors per month per website, with a total of 116,657 unique visitors to all websites. There was substantial variability in website per capita traffic (range 1-167 unique visitors/100,000 Internet users per month; median 31.8, IQR 15.7-47.2).

The websites commonly reported on hospital quality (89%, 16/18), with 61% (11/18) also reporting on clinic or medical group quality (Table 1). There were fewer reports from the Southern region compared to other regions, with reports approximately evenly split through the rest of the regions. Most reports had a state as a catchment area, with all websites reporting at the provider level (hospitals, clinics, or medical groups).

Visitors arrived most often through a search engine query (63.95%, 74,606/116,657 of unique visitors) (Table 2) and less often through referral from another website (15.80%, 18,432/116,657) or direct links received in an email or in an electronic document (19.99%, 74,606/116,657). There was a positive association between percent of unique visitors arriving from search engines and total unique monthly visitors per 100,000 Internet-using households in the catchment area, adjusted for report type (hospital only, outpatient group only, or both), with a 1-point increase in traffic for every 1.8% point increase in proportion of search traffic (P=.002) (Figure 1). The sensitivity analysis that allowed clustering by website returned similar results.

For most websites (61%, 11/18), the search term by which visitors arrived most often was the name of the website. However, collectively, searches for individual hospitals by name were the bulk of the searches that led to visits across all participating websites (Table 2). Among the 2 websites with the highest traffic, responsible for 56.55% (65,967/116,657) of all unique visitors, the hospital name was the most common search term (37.00%, 17,205/46,500 and 97.23%, 9891/10,172 of search terms used), and search was the most common source of traffic (89.00%, 46,500/52,247 and 74.14%, 10,172/13,720 of unique visitors, respectively).

Of all unique visitors (N=116,657), 49,560 were presented the option to take the survey, resulting in a view rate of 42.48%. Of those who viewed the invitation, 1755 responded, resulting in a participation rate of 3.54%. The number of responses from the websites ranged from 2-287 (mean 97.5, SD 98.6; median 49.5, IQR: 26-143).

There were more consumer respondents (850/1755, 48.43%) than health care professional respondents (551/1755, 31.39%). The remaining respondents were members of the media, employers or labor union members, researchers, insurers, or others who chose a free-text option (354/1755, 20.17%). There was wide variation across websites in proportions of respondents who were consumers (21%-71%) and health care professionals (16%-48%). Figure 2 displays that variation and shows ranking according to per capita traffic. Figure 2 illustrates that there is no association between audience composition and website traffic (P=.56 for regression of proportion of consumers and per capita traffic, with similar results in the clustered sensitivity analysis).

More health care professionals than consumers had a primary purpose of choosing or comparing providers (38.8%, 168/433 vs 25.3%, 108/427) (Table 3), whereas more consumers than health care professionals (27.4%, 117/433 vs 20.3%, 88/427) had a primary purpose of finding quality information on a specific provider (P<.001 for overall comparison). For consumers with a primary purpose to “choose providers” who were also asked whether they did so (n=51), 78.4% (40/51) were likely or very likely to use the information to choose a provider. Only 4.2% (18/433) of health care professionals said that they came to the websites for the purpose of patient referral to a hospital or other health care provider (Table 3). Sensitivity analysis found that these patterns were similar by report type (data not shown).

Few providers (0.9%, 4/433) or consumers (2.6%, 11/427) had a primary purpose of looking at cost information. The interest in this information was more common, with both providers (26.6%, 115/433) and consumers (20.8%, 92/442) desiring cost content to be added to the websites (Table 3). Approximately one-third of consumer respondents indicated interest in adding measures about diseases relevant to them (36.4%, 161/442) or adding written comments from other patients (31.5%, 139/442). Consumer respondents rated their experiences using the website lower than did health care professional respondents (mean 6.2, SD 2.7 vs mean 7.2, SD 2.2 on a scale 0-10, P<.001).

Sensitivity analysis found that these differences in primary purpose by respondent type and in mean experience scores were similar in analyses stratified by report type (hospital report vs outpatient quality report, data not shown).

Consumer respondents were commonly middle aged (58.1%, 194/334 were 45-64 years), white (84.4%, 217/257), and many had private insurance (74.1%, 238/321). Additional respondent characteristics are in Multimedia Appendix 3.

In this study of Web analytics data and real-time survey data from a multistate group of public reports of quality, we analyze variations in sources of traffic across websites as well as audience type and purpose. We found that overall traffic to the sites is low. Per capita Internet traffic varied extensively across sites, with higher traffic on websites associated with higher proportion of traffic from search engines. Most visitors arrived after a Web search, frequently using search terms for a specific hospital. Although both consumers and health care professionals use the websites to assess provider quality and choose a provider, websites varied substantially in the proportion of consumer or health care professional respondents, and consumer respondents had a less positive experience on average than health care professionals. Our findings speak to 2 mechanisms through which a public report can be influential: achieving a larger audience and meeting the needs of the audience that arrives.

The number of visitors to these websites was low (1440 unique visitors/month) compared to a similar site in the United Kingdom, the NHS Choices website, which posts numerical ratings of hospitals and physicians as well as written comments from patients. NHS Choices has 250,000 page views per month (some of which could be repeat visitors) to the pages that show comparative provider performance [21]. To the extent that public reports influence choice [22], leading to better health, or that public reporting can incentivize providers to improve care [23,24], increasing the use of US public reporting websites will be important. The UK experience implies that it is potentially possible to do so.

In our group of websites, the range of per capita traffic was large, with the 2 websites with the highest per capita traffic achieving most of their traffic via search queries. This, taken with the finding that search traffic was >60% of traffic overall and that a higher proportion of search traffic is statistically significantly associated with higher overall traffic, suggests that increasing search engine traffic is a potentially powerful approach to increase traffic overall. This is supported by data that in 2012-2013 there were 97,000 monthly Google searches using keywords related to hospital quality, suggesting that consumers are actively searching for this information [25].

We also found that very few visits to the websites came from searches for specific diseases. A similar finding in the survey was that few consumer respondents chose “learn about a disease” as their primary purpose for their visit. Taken together, these findings imply that the provision of disease-specific information only (eg, sections titled “learn about your diabetes” without performance measures) may not increase traffic to the site.

Lastly, we found that the 2 websites with the highest search traffic and the highest traffic overall had hospital names as the most common search terms leading to their visits. These websites had also “tagged” (ie, made visible to search engines) the hospital names, whereas the other websites did not. Hence, our data suggest that tagging provider names on a public reporting website may increase the chances of the website being found in a search and lead to more visitors.

The influence of a public report also depends on the types of people who find the report and whether the content meets their needs. Prior work in this area includes assessments of predictors of reported past use of online physician rating websites and willingness to pay for physician rating sites, examining consumer demographics as well as models of underlying factors motivating consumer use of physician rating websites (digital literacy, perceived ease of use of Internet) [12,26,27]. Our findings add to this work by surveying visitors in real time, including visitors to hospital public reporting websites, assessing physician and consumer perspectives separately, and assessing the purpose of website visitors and their areas of interest. These new contributions may be useful to report sponsors who would like to not only understand predictors of use, but also preferences of their current audience in order to meet their needs. If the traffic on these sites grows, report sponsors or researchers could track whether the user composition changes and whether audience preferences change.

We found that consumer survey respondents on these websites were predominantly older, white, and privately insured. The high proportion of privately insured respondents could be due to low response rate among other respondents or due to low numbers of uninsured or publicly insured visitors. Should the latter be true, one implication is that vulnerable populations who have historically received lower quality of care may access quality information for provider choice less often. Prior research found that African American survey respondents were less likely than white survey respondents to report seeing comparative quality information for hospitals and doctors [28], even though once minorities are aware of reports, they are more likely to use the information [28,29]. To the degree that use of quality information drives health outcomes, disseminating the information to vulnerable populations, as some communities are already trying to do [30], is one approach to avoid widening existing disparities. Additional research is needed regarding whether vulnerable populations have a choice of provider and, if so, the most effective way to disseminate and encourage the use of the quality information to this group [11].

Websites vary in the proportion of consumer and health care professional respondents. Berwick and Coye [31] and Hibbard [9] describe differences in consumer and health care professional pathways through which quality reports may stimulate improvement; hence, both audience types are important. Future research could assess for potential differences between the marketing approaches or website features to explain differences in report visitor composition. We did not find a relationship between the proportion of consumer respondents and per capita traffic, implying that the higher traffic sites are not necessarily succeeding in the consumer populations. Our findings on website experience suggest that additional research is needed to improve the experience for both audiences. Our data show that although there is some overlap in interests between consumers and health care professionals, the consumer website experience was worse than the experience of health care professionals. The needs of health care professionals and consumers may be different enough that a single report will not be adequate to effectively drive quality improvement through the separate pathways Hibbard and Berwick and Coye describe, but additional research is sorely needed in this area.

Few survey respondents indicated that the primary purpose of their visit was to obtain cost information, likely reflecting that most websites lacked cost information. However, 21% of consumers and 27% of health care professionals indicated interest in adding cost information. Since we collected our data, the insurance market has evolved with greater use of high deductible plans and more cost shifting to consumers. In this context, consumer interest in costs may increase [1], especially if there is increased awareness of variation among providers in cost of care [32]. Our data suggest that providing cost information on the websites may be an opportunity to meet the needs of visitors to the public reports.

Consumer respondents reported an interest in seeing patient comments. This is similar to prior literature showing that consumers find patient stories to be as persuasive as quantitative assessments of patient experience [33]. Providing patient comments on provider quality may better meet the needs of consumer visitors. Additional research is needed to understand how to best elicit narrative comments and how to display narrative and quantitative data together to facilitate optimal choices [8,11].

The survey participation rate was low (3.6%) leading to a risk of nonresponse bias. This is a known limitation for website surveys in real time [20] and the results represent the only available data from recent visitors to online public reports. The nonresponse bias in our survey might be particularly relevant in the younger age groups who are poorly represented in these data and are often poor responders to Internet and other types of surveys [34]. This is supported by our findings in Multimedia Appendix 3 showing that the consumer respondents were older than the population in the catchment areas who reported using the Internet to research health plans or practitioners in 2011. Additional methods to gather data from consumer groups who do not answer surveys may be necessary to complete our understanding of visitors to public reports. Response bias does not affect the Web analytics data.

Under the ACA, there is new support for health care transparency and a unique opportunity to help consumers choose higher quality providers. If public reports of provider performance and cost are to be effective, consumers and health care professionals need to find them and visitors to the sites need to find what they need there. These new data suggest that online performance reports of physicians and hospitals are not frequently found and, when found, that the website experience can be improved for both health care professionals and consumers. Using specific search engine techniques may garner a larger audience. Developing reports that cover a broader set of medical conditions, that include patient comments, or that provide cost information could enable website sponsors to better meet visitors’ needs.