In recent years, many health care systems have implemented strategies to improve the quality of care [1]; nevertheless, quality deficits and variability still remain [1-6]. In general, patients are unlikely to be aware of the existence of quality differences [7,8]. One reason for this is the limited amount of publicly available information on the quality of health care providers [9]. Therefore, improving the transparency of information about health care provider quality is a major challenge [10]. It is assumed that this will improve overall quality by steering patients toward better-performing health care providers [11,12] and by incentivizing providers to improve quality [8,9,13-15].

Public reporting in Germany is partly regulated by law. Since 2005, German hospitals have had to publish quality reports online to help patients and physicians make informed choices of hospitals. The AQUA Institute has been commissioned to make further improvements in quality assurance [16]. In 2011, 1666 hospitals had to participate in an “external quality assurance” process. Since the beginning of 2012, the quality reports of individual hospitals include 182 of 430 quality measures [16]. A possible quality shortfall at a hospital can trigger an evaluation, including a structured quality dialog, which allows a group of experts to conduct a qualitative investigation of discrepant results at individual hospitals. In 2010, a total of 21,053 discrepant results were identified in 4,064,320 datasets. Of these, 8.0% were evaluated as qualitatively discrepant through the structured quality dialog. The AQUA Institute does not report on individual hospitals, but German public reporting portals draw on the information provided in the quality reports of individual hospitals.

Nevertheless, there are several barriers to effective public reporting in Germany. Friedemann et al [17] analyzed quality reports of individual German hospitals and concluded that they were neither readable nor understandable for most patients. In international studies, 1 of the barriers most frequently discussed is that consumers do not understand the formats in which the information is presented. Hibbard [18] noted that we need to find more effective ways to present data to consumers. Similarly, Sinaiko et al [19] argued that the current report cards need to be substantially expanded and refined. Kullgren and Werner [20] added that the problem with current public reporting is partly caused by the limitations imposed by the design of report cards.

Although the number of public reporting websites is likely to continue to rise [21], many argue that in their current state they might confuse consumers. Rothberg and colleagues [22] even argued that it would be better to report nothing at all rather than misleading information. Similarly, Emmert et al [23] stated that patients or physicians should not yet use such information to choose an individual physician. In the rush to make providers accountable, enthusiasm has often outstripped science [22]. Several researchers have pointed to the tremendous diversity in the presentation of quality data [21]. The websites and related reports vary widely in terms of ease of access, ease of use, usefulness of information, timeliness of updates, and credibility [24].

After reviewing the available international recommendations, we analyzed the 10 most commonly used German public report cards and addressed 3 questions:

We focused on elements of information design that are used to communicate ideas, illustrate information, or express relationships visually, such as pictures, symbols, and colors [25].

The study consisted of 5 steps: (1) identification of best-practice evidence about the presentation of information on hospital report cards, (2) selection of a single risk-adjusted quality indicator for the study, (3) selection of a sample of the public report card designs used by German hospitals, (4) identification of information presentation elements used in public reporting initiatives in Germany, and (5) conduct of an online-based survey.

A literature search was conducted in April 2013 by searching the Medline (via PubMed) and Cochrane Library databases using the “Abstract/Title/Keywords” search field: Search (“Public Report$ OR Publicly Report$ OR Publicly Release$ OR Public Disclos$ OR Information Disseminat$ OR Report Card$ OR Consumer Report$ OR Quality Report$ OR Comparative Report$ OR Reporting Instrument$”); limitations: English, German, and Spanish; published in the last 10 years; field: “Title/Abstract/Keywords.” Only peer-reviewed journal articles and 3 types of review were included: (1) studies which compared health care report cards in terms of specific criteria, (2) studies which theoretically or empirically assessed or discussed the distinguishing features of public report cards and provided evidence on how performance data should be published or presented, and (3) studies which summarized or discussed best practice in public reporting for health care.

Several risk-adjusted outcome quality measures were available from the German Hospital Quality Report 2011 [17]. The selection of the quality indicator for the study was done by assessing case numbers, the number of hospitals using the measure, and its role in quality assurance. We selected an elective procedure because publicly available information on elective treatments might be expected to help patients identify a good health care provider. We selected a risk-adjusted mortality rate (RAMR) measure because these are considered useful indicators of hospital quality [26-28].

To identify relevant hospital websites, we used a Google search by searching on several keywords (eg, Kliniksuche [clinic search], Krankenhaussuche [hospital search], Gute Klinik [good clinic], Klinikvergleich [compare clinic]). The most frequently visited hospital rating websites were identified using the Alexa analysis tool.

The presentations of the selected quality indicators in 10 report cards were captured with screenshots that included interactive features (mouseovers). Information presentation elements were categorized with nVivo 10 by 2 authors using the previously identified literature-derived categories. Additional categories were added when no literature-derived categories were available.

We applied an online-based cross-sectional study by surveying an online panel to address the following questions:

Consultation took place with an online panel in Germany in August-September 2013; each participant received €1 per finished survey. The panelists were recruited through several recruitment channels including online recruitment, direct mailing, and offline recruitment. The panel members were invited by email to participate (the invitation contained a link to the online survey). The survey was administered and conducted by Norstat Germany Ltd (formerly ODC Services Ltd), a fieldwork agency for survey research.

The online questionnaire consisted of 3 parts. First, a short introduction described the background and study objectives. Second, a random selection of 3 of the 10 hospital report cards was presented to prevent training effects (eg, overfamiliarization with presentation techniques leading to biased results). Respondents were asked to select the best quality hospital, to justify their choice in response to an open-ended question, and to assess the comprehensibility of the website (range of 1=very comprehensible to 7=very incomprehensible). Third, respondents provided sociodemographic data. The questionnaire was piloted and descriptive analyses were conducted using SPSS version 21.0 (IBM Corp, Armonk, NY, USA). The statistical significance of differences between responses was calculated using chi-square tests and t tests. Analysis of the open-ended questions was conducted by 2 authors coding and categorizing answers independently using nVivo 10; discrepancies were discussed to achieve a consensus.

Our search yielded 2506 hits in the Cochrane Library and 1827 in PubMed. After elimination of duplicates, 4302 articles were screened by title and abstract, resulting in exclusion of 4018 articles. In addition, 7 studies were identified through reference search, expert consultation, and Internet searches, giving 291 articles for full-text review. Of these, 13 articles published between 2001 and 2013 met the inclusion criteria. Ten articles described observations in the United States, 2 in the Netherlands, and 1 in Germany.

Table 1 shows the literature-derived categories for the presentation of information on health care report cards. Because this study focused on outcome quality measures, only the information presented on these measures is described here. The methodological quality of the literature used is described in Multimedia Appendix 1.

For our investigation, we selected the risk-adjusted quality indicator coronary angiography and percutaneous coronary intervention (PCI). This procedure was performed 715,469 times in 841 German hospitals in 2011. In 2011, 6369 of 276,866 (2.30%) patients died after PCI; 2.24% mortality had been expected, resulting in a RAMR of 1.03 [17].

A total of 63 hospital public reporting websites were identified by a Google search. Several report cards were eliminated because they did not present outcome quality measures or used presentation formats identical to those of sites already included in the sample. Of the remaining report cards, the 10 most frequently visited were used as a sample (Multimedia Appendix 2): Portal A [39], B [40], C [41], D [42], E [43], F [44], G [45], H [46], I [47], and K [48].

The formats used to present information about RAMR in coronary catheterization by the 10 public reporting websites (see Multimedia Appendix 2) that we studied are summarized in Table 2. Tables (5 sites) and bar charts (5 sites) were equally popular; 4 sites presented reports with incomplete or missing data (“N/A” as a value). Symbols such as traffic lights were commonly used (7 sites), sometimes in combination with bar charts (4 sites) or tables (5 sites) (see Figure 1). Five report cards used low values (for the mortality rate) to indicate good performance and we identified 2 report cards which indicated a “good quality” range for the RAMR.

In total, 3064 respondents started the online survey and 2027 completed it (completion rate=66.16%), taking a mean 14.51 (SD 9.39) minutes. The overall mean age of respondents was 41.57 (SD 15.87) years; 978 of 2027 respondents were female (48.24%) and 50.71% (1028/2027) graduated from high school or obtained a technical university entrance qualification (see Table 3). A total of 96.69% (1960/2027) used the Internet at least once a day. Table 3 also shows the results of the Arbeitsgemeinschaft Online Forschung (AGOF) Internet Facts 2014-07 survey of the German population who used the Internet in the past 3 months [49]. Comparing our survey results to those of the AGOF, the strongest difference appears to be in the demographics of the 2 surveys, specifically in the rate of persons without school qualifications or with secondary general school. In our survey, this rate was 11.69% (237/2027) compared to 35.2% in the AGOF survey. Differences in age, gender, and household size were weaker.

Table 4 shows the results of the survey. Three of the 10 report cards were presented to each respondent (N=2027) for a total of 6081 observations. For each site, a mean 60.68% (1230/2027) of respondents successfully identified the hospital with the lowest RAMR, and 6.81% (138/2027) (range 0.8%-14.0%) selected the hospital with the highest RAMR. In approximately 14.60% (296/2027) of all observations, respondents stated they were unable to provide an answer. Only 32.02% (649/2027) of respondents selected the hospital with the lowest RAMR on all 3 randomly assigned report cards, whereas 14.01% (284/2027) did not identify the hospital with the lowest RAMR on any report card.

The 3 research questions answered are as follows:

We identified 10 elements that were used in the presentation of RAMR for coronary catheterization by 10 German portals (see Table 2).

Report cards using the following 7 presentation elements were more comprehensible, because respondents choose the hospital with the lowest RAMR significantly more often: (1) bar chart without symbols, (2) bar chart with symbols, (3) bar chart with traffic light symbols, (4) bar chart with thumb symbols, (5) providers ranked by performance, (6) an explicit statement about whether higher or lower values indicated better performance, and (7) no statement about scale direction, but a presented range for good quality.

Furthermore, respondents rated presentations as more comprehensible when they contained the following 4 elements: (1) bar chart with symbols, (2) bar chart with traffic light symbols, (3) providers ranked by performance, (4) explicit statement about whether higher or lower values indicated better performance, and (5) no statement about scale direction, but range for good quality presented.

Report cards using the following 2 presentation elements were less comprehensible, because respondents using these elements choose the hospital with the lowest RAMR significantly less often. This was true for tables without symbols or for incomplete data (N/A labels). Moreover, respondents rated presentations as less comprehensible when they contained tables without symbols or incomplete data (N/A labels).

Based on the 13 identified international reports about the presentation of information on report cards, 14 information presentation elements were identified. These studies relied on a variety of methods, including in-depth or cognitive interviews and focus groups, plus online, telephone, or paper-based surveys as well as controlled laboratory experiments (see Multimedia Appendix 1). This variety of methods made it possible to obtain insights from different perspectives about the behavior of users, but also made it more difficult to systematically compare the results. Additionally, samples used in the studies varied widely; sizes ranged from 59 to 2052. Sample selections also had limitations, such as the use of convenience samples, limited geographic locations, low response rates, or a possible selection bias because of questioning by mail and Internet. These limitations have to be taken into account when conclusions are drawn from those studies.

Our respondents, like those in the study by Gerteis et al [29], made more interpretive errors with the numeric formats (table without symbols) than with the graphical formats (bar charts). They had more difficulty understanding numeric tables than other presentation formats, a finding similar to that of Donelan et al [31].

Similar to Geraedts et al [30], we observed that the hospital with the lowest mortality was chosen more often when tables with symbols were used than when tables without symbols were used. In our survey, tables with symbols were rated better for comprehension (mean 3.58, SD 1.84) than bar charts without symbols (mean 2.99, SD 1.84). However, we cannot fully endorse Gerteis et al’s [29] recommendation (in a study limited to 90 respondents) that tables with symbols should be used rather than a standard bar chart without symbols. In our survey, bar charts without symbols more frequently resulted in the choice of the hospital with the lowest mortality (64.47%) than tables with symbols (51.93%), but as discussed previously, missing data rather than the presentational format may have been the major reason.

Similar to Damman et al [32], we found that bar charts were used quite often and, like Gerteis et al [29], we found that graphical formats (standard bar chart without symbols) were not liked by respondents. However, we did not confirm the findings of Geraedts et al, who presented the formats of presentations to physicians, that bar charts without symbols did not assist the comprehension of data on hospital quality [30] or were the format least well understood by participants [29]. In our sample, more respondents who used a bar chart without symbols (64.47%) chose the hospital with the lowest mortality rate than respondents who did not (60.33%). As Figure 3 shows, Portal G displayed bars that were too narrow for evaluation, but numbers were also included. Thus, respondents may still have been able to evaluate the presentation by evaluating the numerical value instead of using the bar chart (“It has the lowest value, but I find the charts very difficult to understand”).

The respondents in our sample who used portals that displayed bar charts with symbols were more likely than average to choose the hospital with the lowest mortality rate, a result which is in accordance with other studies [31-33]. Two portals ranked hospitals by performance, as recommended in several studies [12,15,30,33,35,36]. This was also valued by our participants.

Our results support the recommendation that whether high or low values indicate good performance should be explicitly stated [8,36]. Incomplete data (N/A given as a value) had a negative influence on provider assessment, as Gerteis et al [29] suggested.

To summarize, 5 of 14 information presentation elements were found to be associated with better comprehension as reported in literature: (1) bar charts with symbols, (2) explicit statement about whether higher or lower values indicated better performance, and (3) providers ranked by performance. A reduction of user comprehension was associated with (4) incomplete data (N/A labels) and (5) tables without symbols, again in accordance with literature-based evidence. Reasons have been given for the 2 discrepant findings on (6) bar charts without symbols (improvement of comprehension observed in our German survey, but not described in international literature) and (7) tables with symbols (improvement of comprehension described in international literature, but not observed in our German survey).

To our knowledge, this is the first study to systematically analyze the most commonly used public report card designs in Germany. It is also the first study to analyze German report cards using real-world applications [15] instead of controlled laboratory studies. The best-practice evidence found in 13 international studies led to 14 findings about information presentation elements. However, due to limitations of these studies, conclusions have to be drawn carefully. Five of these findings were in agreement with our findings about German report card designs: (1) avoid tables without symbols, (2) include bar charts with symbols, (3) state explicitly whether high or low values indicate good performance or provide a “good quality” range, (4) avoid incomplete data (N/A given as a value), and (5) rank hospitals by performance. However, due to limitations of our study as described subsequently, these recommendations are preliminary and should be subject to further evaluation.

The implementation of 4 of these findings should not present an insurmountable obstacle to public reporting instruments because they can be achieved by redesigning the format. However, ranking hospitals by performance may present substantial difficulties because ordering by performance is often resisted by providers [7]. Ordering makes report sponsors responsible for determining what constitutes a meaningful difference in mortality rates. The benchmarking information available in Germany might provide a basis for ranking providers. However, ranking might also require measures of statistical significance to be provided that have the potential to confuse users [8,31,36,50,51].

Externally validated measures of hospital quality could also be used as the basis for ranking hospitals and are readily available in Germany [17]; indeed, most German portals use these results to assign their traffic light symbols. However, because only approximately 0.1% of quality measures are finally validated as being qualitatively discrepant, the ranking of hospitals in these terms is difficult. As a respondent put it: “All the symbols for hospital quality are green.”

Several limitations are relevant for this study. This report only investigated the design of report cards. Other important features of report cards (eg, the quality of explanations, use of technical terms, and provision of a quality framework) will be the subject of further studies. Respondents in our study only saw a small part of the public reporting portals because we focused on the presentation of 1 RAMR measure. Five report cards (Portals B, E, F, G, I) did not allow a performance comparison of hospitals in the way we presented it in this study, but were adapted to enable respondents to make the comparison we were investigating. We limited our search for recommendations on best practice in the presentation of information in public report cards to the PubMed and Cochrane medical databases and did not take into account other databases, such as PsychINFO.

Because our study was designed as a Web survey, the results presented might be influenced by self-selection of the study participants. Better-educated persons were overrepresented. Because education levels influence comprehension of public report cards, this might have influenced our results; certain features of presented information might have been misunderstood depending on the education of respondents. Hibbard et al [52] stated that higher education levels were related to improved performance. Emmert et al [53] found that those with a higher level of education tended to comprehend public reporting better. Damman et al [32] stated that consumers’ educational level was positively related to the correct interpretation. Donelan et al [33] found that respondents with at least some college education were significantly more likely to identify surgeons with the lowest risk-adjusted mortality, compared with respondents having no college education.