Published on in Vol 23 , No 6 (2021) :June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/17095, first published .
Rating Hospital Performance in China: Review of Publicly Available Measures and Development of a Ranking System

Rating Hospital Performance in China: Review of Publicly Available Measures and Development of a Ranking System

Rating Hospital Performance in China: Review of Publicly Available Measures and Development of a Ranking System

Original Paper

1School of Public Health, Shanghai Jiao Tong University School of Medicine, Shanghai, China

2Health Services Management Centre, University of Birmingham, Birmingham, United Kingdom

3Center for Health Technology Assessment, China Hospital Development Institute, Shanghai Jiao Tong University, Shanghai, China

4China Hospital Development Institute, Shanghai Jiao Tong University School of Medicine, Shanghai, China

*these authors contributed equally

Corresponding Author:

Guohong Li, PhD

China Hospital Development Institute

Shanghai Jiao Tong University School of Medicine

227 South Chong Qing Road

Shanghai, 200025

China

Phone: 86 21 63846590

Email: guohongli@sjtu.edu.cn


Related ArticleThis is a corrected version. See correction statement in: https://www.jmir.org/2021/6/e31370

Background: In China, significant emphasis and investment in health care reform since 2009 has brought with it increasing scrutiny of its public hospitals. Calls for greater accountability in the quality of hospital care have led to increasing attention toward performance measurement and the development of hospital ratings. Despite such interest, there has yet to be a comprehensive analysis of what performance information is publicly available to understand the performance of hospitals in China.

Objective: This study aims to review the publicly available performance information about hospitals in China to assess options for ranking hospital performance.

Methods: A review was undertaken to identify performance measures based on publicly available data. Following several rounds of expert consultation regarding the utility of these measures, we clustered the available options into three key areas: research and development, academic reputation, and quality and safety. Following the identification and clustering of the available performance measures, we set out to translate these into a practical performance ranking system to assess variation in hospital performance.

Results: A new hospital ranking system termed the China Hospital Development Index (CHDI) is thus presented. Furthermore, we used CHDI for ranking well-known tertiary hospitals in China.

Conclusions: Despite notable limitations, our assessment of available measures and the development of a new ranking system break new ground in understanding hospital performance in China. In doing so, CHDI has the potential to contribute to wider discussions and debates about assessing hospital performance across global health care systems.

J Med Internet Res 2021;23(6):e17095

doi:10.2196/17095

Keywords



Hospital rating systems have the potential to play an important role in patient decision-making as well as offer policy makers and practitioners valuable opportunities to monitor and improve the quality of hospital services [1-4]. In China, significant emphasis and investment into health care reform since 2009 has brought with it increasing scrutiny of its public hospitals with regard to improving their quality and efficiency. Reform measures have included an emphasis on improving hospital governance with clearer regulations and transparency regarding overall performance [5]. Although these measures show promising signs, questions remain about their overall impact and sustainability [6], as well as those concerning the information asymmetries that exist for patients and providers that limit the market conditions of competition and choice deemed necessary to rate hospital performance [7].

An enduring feature of China’s health care provision is the dominance of the hospital sector. Within these contexts, patients are offered different forms of provision ranging from grade I community hospitals, grade II secondary or county hospitals serving several communities, and grade III tertiary hospitals serving districts or cities. This classification [8] remains a powerful driving force for decision-making, with tertiary hospitals often deemed the preferred option for better clinical quality. Pan et al [7] explain how such trends are driven by a culture where patient volume often represents the primary measure of hospital performance used by government administrators. Patients often equate hospital size as a signal of quality, thus preferring to self-refer to larger tertiary hospitals. Large patient volume is also deemed essential for hospitals in developing a good reputation and acquiring high-quality research and training programs.

U.S. News & World Report’s Best Hospitals ranking is one of the well-known hospitals ranking systems that aims to help patients find professional medical centers and doctors across the United States. The relative success of the Best Hospitals ranking demonstrates that the objectivity of measures such as mortality and morbidity can provide an important contribution for accurate evaluation of health care quality [9]. However, influential rankings in developed countries, such as Best Hospitals ranking and Vizient Award [10], are based on solid medical information supporting mechanisms and are challenging to be applied to low- and middle-income countries or regions with relatively underdeveloped medical information supporting facilities.

In order to try and disentangle these trends, China is increasingly seeing the development and use of hospital performance rankings. The annual publication of the Hospital Management Institute of Fudan University Hospital Ranking list [11] ranks hospitals according to a social reputation score that is determined based on survey responses from physicians combined with a review of scientific research outputs from their affiliated institutions. The Science and Technology Evaluation Metrics (STEM) of hospitals developed by the Chinese Academy of Medical Sciences [12] ranks tertiary hospital performance based on their science and technology investment and any associated outputs. Other influential rankings include the top-100 China Hospitals Competitiveness by Alibi Hospital Management Research Center, Hong Kong [13], and China’s best clinical discipline rankings released by Peking University [14].

These indicators provide a valuable contribution to debates and decision-making about hospital performance in China. Nevertheless, given the current situation in China, and the asymmetries of information that exist, important limitations of the current ranking systems have been highlighted, including a reliance on reputation scores [11], a limited menu of performance measures, and a lack of consideration and engagement with measures of quality and safety [8].

Current ranking systems undoubtedly have merit in signifying attempts to better understand hospital performance; however, further research is needed to better understand and triangulate publicly available hospital performance information. Thus far, there has yet to be a comprehensive analysis of what performance information is actually available in China [15]. This study aims to review the publicly available performance information with the view to assess different ways in which hospital performance can be ranked. In doing so, in this paper, we present a new hospital ranking system, termed the “China Hospital Development Index” (CHDI). Although this ranking system faces notable limitations, we argue that our review of measures and development of a ranking system break new ground that can inform both current and future policy and practice for hospital performance in China.


Limitations of Major Hospital Ranking Systems in China

Current hospital performance rankings in China [11-13] classify hospital performance across a range of indictors, including the availability of hospital facilities, services, and personnel; the calculation of social reputation scores; and the publication of scientific research inputs and outputs. These indicators provide valuable contributions for understanding hospital performance; however, a notable limitation in the rankings produced so far has been the emphasis on the quality and safety of health care provision. Quality and safety represent core domains of medical services; therefore, any assessment of hospital performance should aim to incorporate any available measures [16].

It is worth pointing out that for the clinical disciplines ranking reported in Table 1, more than 48 million clinical data records were collected from nearly 400 hospitals across China from 2006 to 2014. The main characteristic of this ranking is that the focus has been shifted to the clinical specialties rather than the number of funds and articles published; it is also the first application of effective medical clinical data for evaluation of hospitals. It should be mentioned that its methodology has not been made public. However, this ranking was unsuccessful (published only once in 2015). The main reason is the standardization of clinical data, such as inconsistent disease codes, which directly affects the quality of medical record information used. Although hospitals in China are vigorously promoting medical informatization at this stage, there is still a long way to use medical data to rank hospitals even if such data are available.

Table 1. Overview of China’s hospital ranking systems.
CharacteristicBest Hospitals ranking [11]STEMa [12]Hospital competitiveness ranking [13]Best Clinical Disciplines ranking [14]
Primary objectiveTo provide guidelines for patients seeking
treatments
To measure a hospital’s value in scientific researchTo identify the top hospitals with the best competitivenessTo provide a tool to help patients find skilled specialty care
Domains of measure
  • Social reputation
  • The ability of sustainable development (scientific research outcomes)
  • Scientific research inputs, outputs, and impacts
  • Medical service
  • Academic impacts
  • Resource management
  • Hospital operation
Unclear
Indicators
  • Social reputation scores
  • SCIb papers
  • National awards
  • Key laboratories and projects
  • Researchers
  • Clinical trials
  • SCI papers
  • Medical standards
  • Medical association leaders
  • National awards
  • Inpatients and outpatients
  • Beds
  • Health workers
  • Medical facilities
  • Personnel
  • Medical fee
  • Length of stay
  • Academic leaders
  • Key laboratories and projects
  • National awards
  • Perioperative mortalities
  • Readmissions
  • Postoperative complications
  • Timely care
  • Finance
Data sourcesNational surveysSCI database and official documentsReporting data voluntarilyMedical records
Publication frequencyAnnuallyAnnuallyAnnuallyOnly in 2015
Transparency of methodologyProvidedProvidedProvidedNot provided

aSTEM: Science and Technology Evaluation Metrics.

bSCI: Science Citation Index.

Exploring Available Performance Measures

Based on the assessment of current measures and the identification of areas for improvement, a group of 6 experts with physician and methodological expertise in performance measurement was established within the China Hospital Development Institution (HDI) to assess the available options and provide feedback at each step of the process (see Multimedia Appendix 1). To begin the analysis, we mapped publicly available measures and identified different information sources that reflected our interest in better understanding hospital performance. Through iterative discussions among the study group, a review of the available literature, and discussions with experts, we established three performance domains for the purpose of ranking hospitals in China. These domains were categorized as research and development, academic reputation, and quality and safety (described below and summarized in Table 2).

Table 2. Summary of measures and clustering of performance indicators into three domains: research and development, academic reputation, and quality and safety.
Domains and measuresIndicatorsData sources
Research and development

Publication outputsNumber of SCIa papers by authors, by first author, and by corresponding authorWeb of Science [17]

Number of citationsNumber of citations by authors, by first author, and by corresponding authorWeb of Science [17]

Number of high-impact outputsNumber of SCI papers with journal IFb ≥10 by authors, by first author, and by corresponding author; number of papers published in top-6 journals by authors, by first author, and by corresponding authorWeb of Science [17]

Clinical trial activityNumber of registered clinical trialsChiCTRc [18]
Academic reputation

AcademicianNumber of academicians of CASd and CAEeCAS [19], CAE [20]

Chief editorNumber of staff as chief editors of core medical journals included in CSCDfCSCD, Science China [21]

Association chairperson or memberNumber of staff as National Association chairperson and National Association membersCMAg [22], CMDAh [23]

AwardSPSTAi, SNSAj, STTPAk, CMSTAl, CDAmCMA [22], CMDA [23],
NOSTAn [24]
Quality and Safety

Quality of specialty careNumber of national key clinical specialties,
diagnosis and treatment, Improvement of Rare Diseases Program
Official websites of each hospital; NHCo, People’s Republic of China [25]

Medical malpractice claimsRatio of compensation cases,
ratio of liability
Laws and Regulations – Peking University [26]

aSCI: Science Citation Index.

bIF: impact factor.

cChiCTR: Chinese Clinical Trial Registry.

dCAS: Chinese Academy of Sciences.

eCAE: Chinese Academy of Engineering.

fCSCD: China Science Citation Database.

gCMA: Chinese Medical Association.

hCMDA: Chinese Medical Doctor Association.

iSPSTA: State Preeminent Science and Technology Award.

jSNSA: State Natural Science Award.

kSTTPA: State Scientific and Technological Progress Award.

lCMSTA: Chinese Medical Science and Technology Award.

mCDA: Chinese Doctor Award.

nNOSTA: National Office for Science and Technology Awards

oNHC: National Health Commission.

To begin our analysis of available measures, we used the Health Statistics Yearbook issued by the National Health Committee to gather baseline information regarding outpatient, inpatient, and emergency admissions to hospitals in China [27]. Second, given the importance placed on research and development in China as a measure of performance, we also sought to identify research and development indicators for hospitals gathered from research databases in order to gauge the research activity and outputs being produced by each hospital. This would include any hospital affiliation of authorship to published Science Citation Index (SCI) papers, the number of citations obtained, and the number of SCI papers for which an impact factor (IF) ≥10 per hospital. Information regarding clinical trial activity was also collected as an indicator for research activity.

As a follow-up to the research and development activity, we were able to obtain measures demonstrating clinical academic reputation of hospital staff engaged in high-impact research outputs demonstrating wide scholarly impact in their clinical area of expertise. This included measuring the number of academic affiliations with the Chinese Academy of Science (CAS) and the Chinese Academy of Engineering (CAE); the number of staff as chief editors of core medical journals included in the China Science Citation Database (CSCD); membership of national associations, including the Chinese Medical Association (CMA) and the Chinese Medical Doctor Association (CMDA); and the number of national awards received per hospital, including the State Preeminent Science and Technology Award (SPSTA), the State Natural Science Award (SNSA), the State Scientific and Technological Progress Award (STTPA), the Chinese Medical Science and Technology Award (CMSTA), and the Chinese Doctor Award (CDA).

Finally, our analysis of the quality and safety performance measures was able to draw on medical malpractice litigation records [28,29] adjusted for complexity and risk of patient disease as two useful indicators for patient safety. Based on the experience the team had in analyzing litigation data as a measure of quality, the selection of such measures resonates with others such as Wang et al [28], who argued that in the absence of more robust indicators, records of medical malpractice litigation in China warranted further exploration as an indicator of health care quality. Additional measures of clinical quality were accessed by reviewing hospital standards and accreditation of treatment excellence performance against the National Health Commission’s Diagnosis and Treatment Improvement of Rare Diseases Program and National Key Clinical Specialty Program.

Developing a Hospital Ranking System

Several rounds of expert consultation identified publicly available indicators, deliberated their utility, and assessed how best to triangulate and weight these measures into comparative performance information. Following the identification and clustering of the available performance measures, we set out to translate these into a practical performance ranking table to assess variation in hospital performance.

Our analysis of operational size and scale highlighted practical limitations to the sample of hospitals included in our ranking. As a result, we focused on the tertiary hospital sector based on the availability of current data as well as to provide an option for comparison with other available measures. By the end of 2017, according to the China Health Statistics Yearbook, there were 1360 grade III, level A hospitals nationwide [27]. The inclusion of these hospitals over others was on the basis that these organizations continue to be the focus of attention in China given their prominence and popularity. These hospitals have also been the focus on other performance rankings in China; hence, the development of any new ranking system would be comparable with other respective performance measures. Our inclusion criteria, therefore, required hospitals to be a grade III, level A hospital, featuring on one of the lists of the four Chinese Hospital rankings in any previous year, and have at least 500 beds. A total of 310 hospitals were thus deemed eligible for ranking under the full criteria.

To develop our ranking system, we relied on statistical procedures such as principal component analysis (PCA) and categorical principal component analysis (CATPCA) [30,31]. PCA is defined as a variable reduction technique that can be used when variables are highly interrelated, providing a way to reduce the number of observed variables into a smaller number of linear, uncorrelated summary variables called principal components (PCs) that account for variation in observed variables. Here, we hypothesized that the various candidate indicators for a given hospital can be represented by several underlying, or latent PCs that reflect the overall strength of this hospital. Thus, for each PC, the model can estimate the extent to which the values are the result of a relationship with the composite score. The remaining variance in the indicators is attributed to measurement error. The degree to which an indicator is correlated with other indicators helps to determine its weight in the equation for the composite scores.

We developed PCA/CATPCA models for each of the three domains of ranking, by evaluating model statistics for all possible combinations of indicators that included at least one indicator. From the resulting list of candidate models showing acceptable fit statistics, we selected a final model for each domain, providing a combination of the number of indicators (models with more indicators produce more accurate component scores), number of outcomes, and model fit.

The PCA/CATPCA replaces the original n features with a smaller number of m features, which are linear combinations of the old features, making these linear combinations as irrelevant as possible. To adjust for linear distribution, before incorporating the values of PCs into the scoring, we implemented a logit transformation for each domain. Equation (1) presents the formula for logit transformation:

Score(i): final score for PCi; i: PCi; G(m): accumulation of variance; Ci: variance for PCi; and Fi: original score for PCi.

Entropy Weight Method

For each domain, we incorporated the values of PCs into the scoring by using entropy weight method, in which weights are systematically calculated based on the level of the difference between the original values. Simply put, if the value difference between the objects, when evaluated using an indicator, is higher than the difference using other indicators, that indicator has more weight than other indicators [32].

Matrix after logical transformation:

where n: number of hospitals; m: number of variables.

Weighting

Deliberations between the expert panel of HDI stakeholders determined the appropriate weights for each domain based on their importance in defining the overall attributes of strength within hospitals.

For presentation purposes, we created what we define as CHDI to measure the development level of the hospitals evaluated. Raw scores were transformed to a scale that assigns a CHDI score of 100 to the top hospital. The formula for the transformation is shown in Equation (7):

CHDI score = (raw score – minimum) / range (7)

Before applying PCA, we also measured the correlation of each variable using Kaiser–Meyer–Olkin (KMO) analysis. If the KMO value is >.7, there is a relatively high level of correlation among variables, and it is thus suitable to use PCA. Similarly, we calculated Cronbach α coefficient before applying CATPCA. CATPCA is an alternative to standard PCA that is particularly useful for data sets consisting of categorical variables (nominal or ordinal) that might be nonlinearly related to each other. CATPCA quantifies categorical variables using optimal scaling, resulting in optimal PCs for the transformed variables. The correlations, shown in Table 3, provide strong evidence of construct validity.

Table 3. Kaiser–Meyer–Olkin (KMO) analysis (Cronbach alpha values) of the three domains of the China Hospital Development Index.
Domain KMO/Cronbach α
Research and development.850
Academic reputation.802
Quality and safety.936

PCA Results

Table 4 shows the results of the analysis, including the number of original indicators, number of selected indicators, number of PCs retained, and accumulation of variance for each domain. The PCA resulted in two PCs, which explained no less than 81% of the variance in the original matrix for each domain.

For research and development, the first principal component (PC1) is highly correlated with the amount of SCI papers and citations, whereas the second principal component (PC2) is highly correlated with high-quality paper measures such as “number of IF≥10 SCI papers by authors” and “number of IF≥10 SCI papers by first author” with correlation coefficients of 0.489 and 0.575, respectively, between the original variables and PCs identified. For quality and safety, PC1 is highly correlated with the medical malpractice claims measures, whereas PC2 is highly correlated with the quality of specialty care measures.

Table 4. Principal component analysis and categorical principal component analysis results of the three domains of the China Hospital Development Index.
DomainOriginal indicatorsSelected indicatorsPrincipal componentAccumulation of variance
Research and development131120.936
Academic reputation9620.818
Quality and safety3320.887

Results of Entropy Weight Method

Table 5 shows PC1 has a higher entropy weight than PC2 for each domain, suggesting that this component has a bigger difference.

The results of our CHDI rankings by score for the top 10 hospitals are shown in Table 6.

Table 5. Results of the entropy weight method for the three domains of the China Hospital Development Index.
DomainPC1a (%)PC2b (%)
Research and development85.414.6
Academic reputation70.229.8
Quality and safety85.114.9

aPC1: first principal component.

bPC2: second principal component.

Table 6. An example of the China Hospital Development Index (CHDI) ranking results.
RankHospitalCHDI
1XH Hospital, Beijing0.9739
2HX Hospital, Chengdu0.9635
3JZ Hospital, Beijing0.9634
4RJ Hospital, Shanghai0.9632
5ZS Hospital, Shanghai0.9527
6HS Hospital, Shanghai0.9400
7ZY Hospital, Hangzhou0.9284
8CH Hospital, Shanghai0.9206
9RM Hospital, Beijing0.9171
10BY Hospital, Beijing0.9137

Validation

Table 7 shows the correlation between scores of domains for 310 hospitals. All of the correlation coefficients between the total score and the score of each domain are above 0.57. The scores of different domains also correlate well among themselves with correlation coefficients higher than 0.50, indicating that the set of indicators is compact and coherent.

Table 7. Correlation coefficients (r) between scores of the three domains for all hospitals evaluated (N=310).
DomainResearch and developmentAcademic reputationQuality and safetyTotal score
Research and development

r10.7880.5770.959

P valuea<.001<.001<.001
Academic reputation

r0.78810.5970.857

P value<.001<.001<.001
Quality and safety

r0.5770.59710.773

P value<.001<.001<.001
Total score

r0.9590.8570.7731

P value<.001<.001<.001

aNot applicable.


Principal Findings

The ability to assess the performance of hospitals in supporting the delivery of high-quality patient care represents a priority for all health care systems [33]. In China, such interest and scrutiny are growing; however, gaining access to relevant performance information remains challenging [34]. Through our analysis of available performance measures, our study aims to contribute to these discussions and debates with a review of hospital performance measures in China. Compared to other health care systems, most notably those in the United States and Europe, what these various measures show are clear limitations in what is currently available to understand hospital performance. For example, our use of research and development indicators and academic reputation as proxy measures for managerial and clinical leadership are exposed to criticisms for their limited connections to day-to-day hospital practice. Our use of litigation data and accreditation standards as proxy measures for hospital quality and safety again have limitations in terms of how far these reflect the clinical quality of hospital care [35]. There is further work to be done regarding how China can develop more clinically focused performance measures that are comparable across hospitals.

Nevertheless, we would argue that our review and subsequent development of a new ranking system breaks new ground in understanding hospital performance in China. Where current hospital rankings in China often rely on reputation scores and investment (input) measures [8,11,12], our review of publicly available measures and their development into a ranking system appears to be the first in opening up the debate for more rigorous and transparent performance information. This is particularly the case for quality and safety performance. Our review and inclusion of litigation data [28,29,36,37] provides a valuable opportunity to assess the comparative performance of quality and safety across hospitals in China.

Thus, this paper contributes to what appears to be a growing body of knowledge that is using innovative and feasible methodologies in data collection and modeling to better understand the performance of public hospitals in China [38].

Based on our analysis, we suggest that further research and policy development is needed to build on these results. Given the practical limitations of securing comparative data and the interest in benchmarking our analysis with existing hospital rankings in China, our sample has focused exclusively on a number of tertiary hospitals. Our rankings reflect the high performance of these organizations compared to that of other hospital and primary care providers; however, we are also mindful of the possible further imbalance this can create in China’s health care system by virtue of acute medical care over primary and community care. The fact that the majority of our highest ranked hospitals are located in Shanghai and Beijing also illustrates important challenges facing access to high-quality hospital care in other parts of China. Such findings support those of Yu et al [38] who have documented how the unevenness of health care resources in China is closely related to a city’s administrative rank and power: the higher the level, the better the resources. Such arrangements are reinforcing investment in high-ranked hospitals at the expense of primary care services. The correlation between the quality and safety domain and the overall hospital performance in our ranking system is slightly lower than that with the other two domains; therefore, more clinical objective measures should be included to increase the influence of this domain.

Therefore, we call on further research and development to access and compare performance measures from within each hospital, including private hospitals, as well as other parts of the health care system, including primary and community care. For this purpose, China could build on the cross-sectional research it has undertaken into mortality trends [39] and nurse staffing levels [40]. Such research has the potential to be scaled up and developed into performance measures translatable across all hospitals and incorporated into our methodology.

We also support further research and development that draws on the views of a range of different stakeholders in terms of what performance measures would be meaningful for patients, public, and health care staff. Given the well-documented challenges facing the doctor-patient relationship in China [41,42], we encourage deliberative events involving a range of stakeholders to discuss what constitutes good performance with the view to developing shared understanding of performance measurement from different perspectives. The Delphi method [43] is one way to do this, and such an approach has been used in other parts of China to good effect. This includes further development of comparative measures for health outcomes and the development of experiential data about how different stakeholders experience the health care received and how they can improve hospital performance as well as other aspects of China’s health care system.

The year 2019 marks the tenth year for China’s goal to deepen the reform of its medical and health system. In 2019, the Chinese government formulated the National Tertiary Public Hospital Performance evaluation index system and unified the collection of performance information across hospitals [44]. The implications of such changes remain to be seen, with the results of these assessments not yet fully disclosed. However, we anticipate this is an important step in developing greater understanding of hospital performance in China. We believe that our review and the newly developed ranking index (CHDI) has an important role to play in shaping such discussions and assessments, particularly in relation to the improvement of quality and patient safety, as well as raising public awareness regarding the information that is available to inform their decision-making.

Conclusions

The reform of China’s health care system has brought with it increasing scrutiny regarding the quality of care delivered to patients. Our analysis presents what appears to be the first review of publicly available performance measures for hospitals in China. In collaboration with an expert panel, in the review, the available measures have been clustered into three performance domains, namely research and development, academic reputation, and quality and safety of hospital care. Furthermore, based our analysis, we have applied these performance measures to a selection of tertiary hospitals in China with the view to better understand their comparative performance. There remain some notable limitations and challenges regarding this performance information; nevertheless, we believe that our review and ranking system break new ground in assessing hospital performance in China. Although further research and development is clearly needed to enhance and refine this performance information, we argue that the proposed hospital development index sets a new and important research agenda for understanding and improving hospital care in China.

Acknowledgments

The authors acknowledge the research institution, university, and the data collectors for their participation and support in this study. This study was supported by Major Key Research Projects in Philosophy and Social Sciences of the Ministry of Education of the People’s Republic of China (grant 18JZD044) and National Natural Science Foundation of China (grant 72074147).

Authors' Contributions

SJD participated in the acquisition of data, design and analysis of the study, and drafted the manuscript. RM was involved in drafting the final manuscript. CSS contributed to the conception and design of the study. MYD and YYX assisted with data collection and statistical analysis of the data. GHL, JS, and XQF prompted discussion on the implications of the study and its results and critically revised the manuscript for important intellectual content. All authors have read and approved the final version of the manuscript and are accountable for all aspects of the work.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Ranking process of China Hospital Development Index.

DOCX File , 34 KB

  1. Leonardi MJ, McGory ML, Ko CY. Publicly available hospital comparison web sites: determination of useful, valid, and appropriate information for comparing surgical quality. Arch Surg 2007 Sep;142(9):863-8; discussion 868. [CrossRef] [Medline]
  2. Hwang W, Derk J, LaClair M, Paz H. Finding order in chaos: a review of hospital ratings. Am J Med Qual 2016;31(2):147-155. [CrossRef] [Medline]
  3. Huerta TR, Walker DM, Ford EW. An evaluation and ranking of children's hospital websites in the United States. J Med Internet Res 2016 Aug 22;18(8):e228 [FREE Full text] [CrossRef] [Medline]
  4. Bilimoria K, Birkmeyer J, Burstin H. Rating the raters: an evaluation of publicly reported hospital quality rating systems. NEJM Catalyst 2019 Aug 14.
  5. Mei J, Kirkpatrick I. Public hospital reforms in China: towards a model of new public management? IJPSM 2019 May 13;32(4):352-366. [CrossRef]
  6. Yip W, Fu H, Chen AT, Zhai T, Jian W, Xu R, et al. 10 years of health-care reform in China: progress and gaps in Universal Health Coverage. Lancet 2019 Sep 28;394(10204):1192-1204. [CrossRef] [Medline]
  7. Pan J, Qin X, Hsieh C. Is the pro-competition policy an effective solution for China’s public hospital reform? HEPL 2016 Jun 27;11(4):337-357. [CrossRef]
  8. Wu S, Chen T, Pan Q, Wei L, Xuan Y, Li C, et al. Establishment of a comprehensive evaluation system on medical quality based on cross-examination of departments within a hospital. Chin Med J (Engl) 2017 Dec 05;130(23):2872-2877 [FREE Full text] [CrossRef] [Medline]
  9. FAQ: how and why we rank and rate hospitals. U.S. News. 2020 Jul 28.   URL: https:/​/health.​usnews.com/​health-care/​best-hospitals/​articles/​faq-how-and-why-we-rank-and-rate-hospitals [accessed 2020-07-30]
  10. Member awards: a different approach for 2020. Vizient.   URL: https://www.vizientinc.com/members/member-awards [accessed 2020-07-30]
  11. Xiong X, Gao J, Zhang B, Lu J, Luo L. Introduction of Chinese hospital ranking method from the aspect of theoretical framework, practical choice and social effect. J Hosp Manag Health Policy 2017 Nov 6;1:4-400. [CrossRef]
  12. Dai T, Qian Q, Xin-Ying AN. Research on the Sci-tech Influence Evaluation Index System for hospitals in China. Article in Chinese. Journal of Medical Informatics 2016;37(3):2-7 [FREE Full text]
  13. Chong E, Yan A, Wang S. Research on the comprehensive ranking method of hospital competitiveness. Article in Chinese. Chinese Journal of Medical Management Sciences 2016;6(3):45-50 [FREE Full text]
  14. Peking University released "China's Best Clinical Discipline Rankings". People's Daily Online. 2015 Jun 4.   URL: http://scitech.people.com.cn/n/2015/0604/c1057-27100408.html [accessed 2021-05-31]
  15. Jian W, Figueroa J, Woskie L, Yao X, Zhou Y, Li Z, et al. Quality of care in large Chinese hospitals: an observational study. BMJ Qual Saf 2019 Dec;28(12):963-970. [CrossRef] [Medline]
  16. Wang DE, Tsugawa Y, Figueroa JF, Jha AK. Association between the Centers for Medicare and Medicaid Services hospital star rating and patient outcomes. JAMA Intern Med 2016 Jun 01;176(6):848-850. [CrossRef] [Medline]
  17. Web of Science. Clarivate.   URL: https://clarivate.com/webofsciencegroup/solutions/web-of-science/ [accessed 2021-05-26]
  18. Chinese Clinical Trial Registry.   URL: http://www.chictr.org.cn/enIndex.aspx [accessed 2021-05-26]
  19. Chinese Academy of Sciences. Webpage in Chinese.   URL: https://www.cas.cn/ [accessed 2021-05-26]
  20. Chinese Academy of Engineering.   URL: http://en.cae.cn/en/ [accessed 2021-05-26]
  21. Chinese Science Citation Database, Science China.   URL: http://www.sciencechina.cn/index.jsp [accessed 2021-05-26]
  22. Chinese Medical Association.   URL: http://en.cma.org.cn/ [accessed 2021-05-26]
  23. Chinese Medical Doctor Association.   URL: http://www.cmdae.org/ [accessed 2021-05-26]
  24. National Office for Science and Technology Awards.   URL: http://www.nosta.gov.cn/english/index.html [accessed 2021-05-26]
  25. National Health Commission of the People's Republic of China.   URL: http://en.nhc.gov.cn/ [accessed 2021-05-26]
  26. Laws and Regulations – Peking University. Webpage in Chinese.   URL: www.pkulaw.cn [accessed 2021-05-26]
  27. China Statistical Yearbook 2018. Beijing: China Statistics Press; 2018.
  28. Wang Z, Li N, Jiang M, Dear K, Hsieh C. Records of medical malpractice litigation: a potential indicator of health-care quality in China. Bull World Health Organ 2017 Jun 01;95(6):430-436 [FREE Full text] [CrossRef] [Medline]
  29. Järvelin J, Häkkinen U. Can patient injury claims be utilised as a quality indicator? Health Policy 2012 Feb;104(2):155-162. [CrossRef] [Medline]
  30. Linting M, van der Kooij A. Nonlinear principal components analysis with CATPCA: a tutorial. J Pers Assess 2012;94(1):12-25. [CrossRef] [Medline]
  31. Meulman J, Heiser W. Categorical principal component analysis (CATPCA). SPSS Categories 2005:14.
  32. Jing N, Wang J. Evaluate the International Talents in Professional Social Networks using the Entropy Weight MethodC//2015 International Conference on Computational Science and Computational Intelligence (CSCI). 2015 Presented at: International Conference on Computational Science and Computational Intelligence; 2015; Las Vegas, NV, USA p. 220-225. [CrossRef]
  33. Li X, Krumholz HM. What does it take to improve nationwide healthcare quality in China? BMJ Qual Saf 2019 Dec;28(12):955-958. [CrossRef] [Medline]
  34. Li C, Xu X, Zhou G, He K, Qi T, Zhang W, et al. Implementation of National Health Informatization in China: survey about the status quo. JMIR Med Inform 2019 Feb 21;7(1):e12238 [FREE Full text] [CrossRef] [Medline]
  35. Garon-Sayegh P. Analysis of medical malpractice claims to improve quality of care: Cautionary remarks. J Eval Clin Pract 2019 Oct;25(5):744-750. [CrossRef] [Medline]
  36. Li H, Wu X, Sun T, Li L, Zhao X, Liu X, et al. Claims, liabilities, injures and compensation payments of medical malpractice litigation cases in China from 1998 to 2011. BMC Health Serv Res 2014 Sep 13;14:390 [FREE Full text] [CrossRef] [Medline]
  37. Greenberg MD, Haviland AM, Ashwood JS, Main R. Is better patient safety associated with less malpractice activity?: Evidence from California. Rand Health Q 2011;1(1):1 [FREE Full text] [Medline]
  38. Yu M, He S, Wu D, Zhu H, Webster C. Examining the multi-scalar unevenness of high-quality healthcare resources distribution in China. Int J Environ Res Public Health 2019 Aug 07;16(16):2813 [FREE Full text] [CrossRef] [Medline]
  39. Zhang Z, Lei J, Shao X, Dong F, Wang J, Wang D, China Venous Thromboembolism Study Group. Trends in hospitalization and in-hospital mortality from VTE, 2007 to 2016, in China. Chest 2019 Feb;155(2):342-353. [CrossRef] [Medline]
  40. Zhu X, You L, Zheng J, Liu K, Fang J, Hou S, et al. Nurse staffing levels make a difference on patient outcomes: a multisite study in Chinese hospitals. J Nurs Scholarsh 2012 Sep;44(3):266-273. [CrossRef] [Medline]
  41. He AJ, Qian J. Explaining medical disputes in Chinese public hospitals: the doctor-patient relationship and its implications for health policy reforms. Health Econ Policy Law 2016 Oct;11(4):359-378. [CrossRef] [Medline]
  42. Hesketh T, Wu D, Mao L, Ma N. Violence against doctors in China. BMJ 2012 Sep 07;345:e5730. [CrossRef] [Medline]
  43. Hu Q, Qin Z, Zhan M, Wu B, Chen Z, Xu T. Development of a trigger tool for the detection of adverse drug events in Chinese geriatric inpatients using the Delphi method. Int J Clin Pharm 2019 Oct;41(5):1174-1183. [CrossRef] [Medline]
  44. Opinions of the General Office of the State Council on strengthening the performance appraisal work of tertiary public hospitals. Webpage in Chinese. Office of the State Council, People's Republic of China. 2019 Jan 30.   URL: http://www.gov.cn/zhengce/content/2019-01/30/content_5362266.htm [accessed 2021-05-25]


CAE: Chinese Academy of Engineering
CAS: Chinese Academy of Sciences
CATPCA: categorical principal component analysis
CDA: Chinese Doctor Award
CHDI: China Hospital Development Index
CMA: Chinese Medical Association
CMDA: Chinese Medical Doctor Association
CMSTA: Chinese Medical Science and Technology Award
CSCD: China Science Citation Database
HDI: Hospital Development Institution
IF: impact factor
KMO: Kaiser–Meyer–Olkin
PC: principal component
PC1: first principal component
PC2: second principal component
PCA: principal component analysis
SCI: Science Citation Index
SNSA: State Natural Science Award
SPSTA: State Preeminent Science and Technology Award
STEM: Science and Technology Evaluation Metrics
STTPA: State Scientific and Technological Progress Award


Edited by G Eysenbach, R Kukafka; submitted 19.11.19; peer-reviewed by W Zhang, R Waitman; comments to author 29.06.20; revised version received 12.08.20; accepted 30.04.21; published 17.06.21

Copyright

©Shengjie Dong, Ross Millar, Chenshu Shi, Minye Dong, Yuyin Xiao, Jie Shen, Guohong Li. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.06.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.