Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 15.07.16 in Vol 18, No 7 (2016): July

This paper is in the following e-collection/theme issue:

    Original Paper

    Influence of Intensified Supervision by Health Care Inspectorates on Online Patient Ratings of Hospitals: A Multilevel Study of More Than 43,000 Online Ratings

    1Radboud University Medical Center, Radboud Institute for Health Sciences, IQ Healthcare, Nijmegen, Netherlands

    2Dutch Health Care Inspectorate (IGZ), Utrecht, Netherlands

    3Academic Medical Center, Amsterdam, Netherlands

    4Radboud University Medical Center, REshape Innovation Center, Nijmegen, Netherlands

    Corresponding Author:

    Rudolf Bertijn Kool, MD, PhD

    Radboud University Medical Center

    Radboud Institute for Health Sciences, IQ Healthcare

    Postbus 9101, Huispost 114



    Phone: 31 611531143

    Fax:31 243540166



    Background: In the Netherlands, hospitals with quality or safety issues are put under intensified supervision by the Dutch Health Care Inspectorate, which involves frequent announced and unannounced site visits and other measures. Patient rating sites are an upcoming phenomenon in health care. Patient reviews might be influenced by perceived quality including the media coverage of health care providers when the health care inspectorate imposes intensified supervision, but no data are available to show how these are related.

    Objective: The aim of this study was to investigate whether and how being under intensified supervision of the health care inspectorate influences online patient ratings of hospitals.

    Methods: We performed a longitudinal study using data from the patient rating site Zorgkaart Nederland, from January 1, 2010 to December 31, 2015. We compared data of 7 hospitals under intensified supervision with a control group of 28 hospitals. The dataset contained 43,856 ratings. We performed a multilevel logistic regression analysis to account for clustering of ratings within hospitals. Fixed effects in our analysis were hospital type, time, and the period of intensified supervision. Random effect was the hospital. The outcome variable was the dichotomized rating score.

    Results: The period of intensified supervision was associated with a low rating score for the hospitals compared with control group hospitals; both 1 year before intensified supervision (odds ratio, OR, 1.67, 95% CI 1.06-2.63) and 1 year after (OR 1.79, 95% CI 1.14-2.81) the differences are significant. For all periods, the odds on a low rating score for hospitals under intensified supervision are higher than for the control group hospitals, corrected for time. Time is also associated with low rating scores, with decreasing ORs over time since 2010.

    Conclusions: Hospitals that are confronted with intensified supervision by the health care inspectorate have lower ratings on patient rating sites. The scores are independent of the period: before, during, or just after the intervention by the health care inspectorate. Health care inspectorates might learn from these results because they indicate that the inspectorate identifies the same hospitals as “at risk” as the patients rate as underperformers.

    J Med Internet Res 2016;18(7):e198




    Patient rating sites (PRSs) are an upcoming phenomenon in health care [1]. In many countries, websites such as RateMDs, Vitals, and Zocdoc in the United States, Jameda in Germany, and NHS Choices in the United Kingdom have created a platform for patients to share their experiences with health care providers. The number of ratings is growing and it comprehends all kinds of care. Nevertheless, the usefulness of PRSs for health care is being discussed [2-5]. Recently, a scoping review showed a growing body of literature on positive relationships between ratings on PRSs and indicators of quality of care such as patient satisfaction, mortality, and readmissions [6,7]. The content of PRSs is used for several purposes including supervision by the Dutch Health Care Inspectorate (Inspectie voor de Gezondheidszorg or IGZ) since 2015 [8].

    Patient reviews are influenced not only by the type of care received by the patient and the way health care is delivered, but probably also by external stimuli such as media coverage. Research has shown that patients, especially the local ones, are influenced in their trust in health care providers by media attention [9]. In the Netherlands, media generally pay close attention to health care providers where “something might be wrong.” Recent examples are 2 hospitals that came under intensified supervision of the IGZ after it noted patient safety problems [10,11]. Intensified supervision means frequent announced and unannounced site visits and consultation with the board, and it will be ended when structural improvements of the quality and safety of care have been proven and the board shows to be in control, see Textbox 1 [12]. In particular, health care magazines and local newspapers and sometimes local or even national radio and television report the intensified supervision and highlight it with special coverage. This could influence the opinion of patients of the health care providers and so their ratings on PRSs.

    The fact that a health care inspectorate uses patient reviews as one of the components in risk detection, and that at the same time patient reviews could be influenced by intervention by the health care inspectorate, raises the question of how patient ratings of health care providers change after publication of intensified supervision by the inspectorate. There is no existing research on the extent to which patient ratings of health care providers are affected by intensified supervision of a health care inspectorate. One might expect that mean patient ratings of health care providers under intensified supervision are generally low compared with other health care providers before intensified supervision, because of the quality problems that are found. Patients and their relatives may use low ratings to draw attention to the problems they have experienced. Our first hypothesis was therefore that health care providers under intensified supervision would have a relative low mean overall patient rating in the period before intensified supervision compared with comparable providers.

    Second, the mean rating of health care providers under intensified supervision might probably increase during the intensified supervision compared with the period before intensified supervision. Although negative media attention can ruin a reputation, previous research has also shown that the trust of patients in their doctors is high [13] and trust influences loyalty [14]. Patients might support their health care provider by sharing positive experiences and higher ratings. Third, we assumed that after the intensified supervision, when most of the publicity is gone, mean ratings of health care providers would be stable and comparable with others.

    To investigate the aforementioned assumptions, we performed a multilevel study with the following research question: To which extent is intensified supervision associated with online hospital patient ratings? Because of the limited number of ratings of other health care providers, we focused on only hospitals in this study.

    Textbox 1. The Dutch Health Care Inspectorate.
    View this box


    Study Design

    We performed an observational study using publicly available data.

    Data Sources

    First, we scanned the website of the IGZ where it publicly announces the providers that will be monitored by intensified supervision in order to arrange a list of all intensified supervision hospitals. We collected the names of these hospitals from January 2010 to December 2015, including the exact start and end date of the intensified supervision. In this period the IGZ decided to monitor 7 entire hospitals intensively by intensified supervision and to end it. We excluded 2 hospitals in which only 1 department was monitored by the inspectorate.

    Second, we used data from the biggest PRS in the Netherlands, Zorgkaart Nederland, with more than 272,000 ratings and more than a million visitors per month in 2015. We used the publicly available data from January 1, 2010 to December 31, 2015, in which period the IGZ did not yet use hospital ratings to decide on intensified supervision. On Zorgkaart Nederland, patients can rate health care providers on 6 aspects, namely, accommodation, appointments, treatment, information, listening, and professionals, based on a scale from 1-10, where “1” stands for “extremely poor” and “10” for “extremely good.” An overall score is automatically calculated (rounded to the nearest 0.1 percentage point). We used the number of ratings, the percentage of reviews with a score lower than 6.5, and the mean rating score on Zorgkaart Nederland given by patients. The 6.5 threshold is based on the theory of the Net Promoter Score [15]. This theory considers the scores 9 and 10 as positive “promoters,” the scores 7 and 8 as neutral, passively satisfied, and the scores 0 to 6 ratings as “detractors,” or negative recommendations. Because we were primarily interested in the “detractors,” as these ratings might tell something about patient safety, we dichotomized the dependent variable rating scores: scores higher than and equal to 6.5 were labeled “0” and scores lower than 6.5 were labeled “1.”


    We compared the data of the 7 intensified supervision hospitals with a control group of 28 hospitals. In the Netherlands, 3 types of hospitals exist: relatively small general acute care hospitals, the bigger teaching hospitals, and the major academic hospitals. The control group hospitals were purposively sampled in the same region (North, South, East, and West) and same type as the intensified supervision hospitals and also on having the most ratings, thus guaranteeing a sufficient number of ratings. The total number of hospitals included in the dataset was 35 with 43,856 ratings. We analyzed the data for the different categories of hospitals because we could expect a difference in rating scores; patient satisfaction does differ between small and major hospitals [16]. We also analyzed the data for every year in order to visualize time effects.

    Because we expected ratings to be influenced quite a while before the intensified supervision, we categorized the intensified supervision period into 5 categories: the period before 1 year before intensified supervision, the 12 months before intensified supervision, during intensified supervision, the 12 months after intensified supervision, and the period after 1 year after intensified supervision. The period during intensified supervision varied according to the decision of the inspectorate to prolong intensified supervision (3-12 months; mean 7 months).

    We performed a longitudinal logistic regression analysis (mixed model) to analyze whether periods of intensified supervision lead to lower patient ratings. To account for clustering of ratings within hospitals, “hospital” was included in the model as the random effect. The outcome variable was the dichotomized rating score.

    Determinants in our analysis were the period of intensified supervision, hospital type, and the course of ratings over time (variable “time” in years). We included the factor time to analyze time trends that occur anyway, irrespective of intensified supervision.

    Ethical Approval

    No ethical approval was needed because we used publicly available data and no persons were directly involved.


    Table 1 lists the numbers of ratings for intensified supervision hospitals and control group hospitals per period and per hospital type. Table 2 presents the mean rating score and percentage of ratings lower than 6.5 per period for intensified supervision hospitals and control group hospitals. Table 3 presents the mean rating in time and the percentage of ratings lower that 6.5, for both groups.

    Table 1. Numbers of rating scores per hospital type and period.
    View this table
    Table 2. Mean rating score and percentage of ratings <6.5 per period for both intensified supervision and control group hospitals.
    View this table
    Table 3. Mean rating in time and percentage <6.5 for both intensified supervision and control group hospitals.
    View this table

    Table 4 presents the results of the multilevel analysis. The period of supervision is associated with a low rating score for intensified supervision hospitals compared with control group hospitals: both 1 year before intensified supervision (odds ratio, OR, 1.67, 95% CI 1.06-2.63) and 1 year after intensified supervision (OR 1.79, 95% CI 1.14-2.81) differ significantly. For all periods the odds on a low rating score for intensified supervision hospitals are higher than for the control group hospitals, corrected for time. The proportion of low rating scores decreased over time since 2010.

    Table 4. Effect of intensified supervision period on low patient ratings, adjusted for time and type of hospital.
    View this table


    On the basis of the results of this study, we can confirm the hypothesis that the average rating of intensified supervision hospitals before the intensified supervision started is lower compared with the control group. We found also that patient ratings were low not only before but also during and just after the intensified supervision: the scores are continuously relatively low. It seems that only during the period after a year after intensified supervision the ratings are comparable with the control group hospitals. This result might assure the inspectorate that intensified supervision does not influence the patient ratings that it uses for supervision significantly in the short term. The results do not indicate that the mean rating will increase during the intensified supervision, for example, because of the loyalty of patients.

    Health care inspectorates might learn from these results because they indicate that the inspectorate identifies the same hospitals as “at risk” as the patients rate as underperformers. This can be seen as another indication of the opportunities for patients to identify patient safety problems [17,18]. Monthly monitoring of scores on PRSs by health care inspectorates or other quality monitoring organizations could be of additional value in identifying health care providers at risk. This is also in line with the results of several studies that show correlations between patient rating scores and quality indicators, although the correlations were mostly weak and sometimes inconsistent [6]. The potential contribution of patient rating scores to health care governance supports the initiatives of health care inspectorates already using these online scores in their daily supervision [8].

    Although not part of the research question, we identified a trend in patient rating scores. Compared with the start in 2010, the mean overall ratings on PRS Zorgkaart Nederland have increased significantly in 2015. To the best of our knowledge, there has not been an analysis of the rating trend over several years. This might be due to relative low ratings of pioneers at the start of the PRS. Now that the PRS is used more, it might attract a broader public with more positive ratings in general. In total, 92% of the ratings in our dataset are positive (≥6.5), which is comparable with approximately 90% of ratings found in other research [19].

    Strengths and Limitations

    An important strength of this study is the major database used with 43,856 online ratings of 35 hospitals, more than a third of all Dutch hospitals.

    In addition, it takes into account the opportunities of using patient experiences in supervision of health care. Especially with the increasing use of rating sites in the near future when the generation socialized with social media (eg, Facebook and Twitter) reaches the age in which health questions and doctors become significant, these kinds of sources might become even more relevant for patients, physicians, and other stakeholders such as health care inspectorates [2,4,20]. An increasing number of patients are given a voice to their experiences in order to identify patient safety risks. Health care inspectorates all over the world might benefit from this and better involve citizens in health care governance.

    A limitation of the study is that we only selected the hospitals with sufficient reviews in the control group. We cannot rule out that this was a selected group of hospitals. Furthermore, the number of hospitals with intensified supervision was low. There were only 7 hospitals confronted with intensified supervision. However, we analyzed the data on the level of patient ratings, which provided us with a dataset with enough power (n=43,856). The number of hospitals under supervision is only a small part of all providers in the Dutch health care with intensified supervision. In 2013 and 2014 it was concerned with 47 health care organizations of which 3 were hospitals. A second limitation is that at the moment the number of ratings of the other health care providers, mainly long-term elderly care, is too low to use for analysis. However, this could change rapidly. The branch organization of long-term elderly care announced in July 2015 the decision to cooperate with the Dutch Patient and Consumer Federation, the owner of Zorgkaart Nederland, in order to increase the number of reviews substantially by collecting ratings via interviews.

    Future Research

    We expect that an increasing number of patients will share their experiences on PRSs and a growing number of patients will use those experiences in their judgment and choice of health care providers. This study suggests that all those experiences could be useful in estimating the quality of care because of the interesting association with the judgment of health care inspectors. However, it takes extensive research to understand this relationship better. Because of this association, this study might encourage health care inspectorates in experimenting with civilians as layman inspectors. It would also be interesting to know what underlying aspect of the IGZ's decision the negative patient reviews may be correlating with.

    Exploring the reasons for low patient ratings in general will be necessary to give hospitals insight into how to improve their ratings. This could be investigated by closely studying the texts of the reviews, for example, by using Web-based text processing tools [21]. It might also be instructive for hospitals and IGZ to explore if any of the 6 aspects that go into the overall patient rating on Zorgkaart Nederland are more specifically correlated to intensified supervision rather than the overall score.

    It might also be useful to explore the use of online patient ratings by health care parties other than inspectorates, for example, health insurance companies. They could start using rating scores to select preferred providers in their purchase of care.

    Finally, research on the influence of supervisory activities should also be performed with other health care providers than hospitals, such as long-term care institutions, under the condition of sufficient number of reviews. Moreover, it would be relevant to repeat the study in other countries to investigate whether a different system of supervision or the presence of several PRSs influences the results.


    Hospitals that are confronted with intensified supervision by the health care inspectorate have lower ratings on PRSs. Health care inspectorates might learn from these results because they indicate that the inspectorate identifies the same hospitals as “at risk” as the patients rate as underperformers. More research with more ratings also in other parts of health care and other countries is needed to explore further the association between ratings on PRSs and the quality judgment of a health care inspectorate.


    We would like to thank the Dutch Patient and Consumer Federation (NPCF) for making available the data from Zorgkaart Nederland. Funding for this project was received from NWO/ZonMw.

    Authors' Contributions

    RBK designed the study and drafted the manuscript. SMK and FA analyzed and interpreted the data. All authors reviewed and revised the manuscript at various stages to its final version and have read and approved the final manuscript.

    Conflicts of Interest

    None declared.


    1. Burkle CM, Keegan MT. Popularity of internet physician rating sites and their apparent influence on patients’ choices of physicians. BMC Health Services Research 2015 Sep 26;15:416. [CrossRef]
    2. Greaves F, Ramirez-Cano D, Millett C, Darzi A, Donaldson L. Harnessing the cloud of patient experience: using social media to detect poor quality healthcare. BMJ Qual Saf 2013 Mar;22(3):251-255. [CrossRef] [Medline]
    3. Galizzi MM, Miraldo M, Stavropoulou C, Desai M, Jayatunga W, Joshi M, et al. Who is more likely to use doctor-rating websites, and why? A cross-sectional study in London. BMJ Open 2012;2(6):e001493. [CrossRef]
    4. Reimann S, Strech D. The representation of patient experience and satisfaction in physician rating sites. A criteria-based analysis of English- and German-language sites. BMC Health Serv Res 2010 Dec 7;10:332 [FREE Full text] [CrossRef] [Medline]
    5. Strech D. Ethical principles for physician rating sites. J Med Internet Res 2011;13(4):e113 [FREE Full text] [CrossRef] [Medline]
    6. Verhoef LM, Van de Belt TH, Engelen LJ, Schoonhoven L, Kool RB. Social media and rating sites as tools to understanding quality of care: a scoping review. J Med Internet Res 2014;16(2):e56 [FREE Full text] [CrossRef] [Medline]
    7. Emmert M, Adelhardt T, Sander U, Wambach V, Lindenthal J. A cross-sectional study assessing the association between online ratings and structural and quality of care measures: results from two German physician rating websites. BMC Health Services Research 2015 Sep 24;15:414.
    8. van de Belt TH, Engelen LJ, Verhoef LM, van der Weide MJ, Schoonhoven L, Kool RB. Using patient experiences on Dutch social media to supervise health care services: exploratory study. J Med Internet Res 2015 Jan;17(1):e7 [FREE Full text] [CrossRef] [Medline]
    9. van der Schee E, de Jong JD, Groenewegen PP. The influence of a local, media covered hospital incident on public trust in health care. Eur J Public Health 2012 Aug;22(4):459-464 [FREE Full text] [CrossRef] [Medline]
    10. Sheldon T. Dutch microbiologists are disciplined for delays during pneumonia outbreak. BMJ 2012 Jan 30;344:e755.
    11. Sheldon T. Senior managers resign over conflict with doctors. BMJ 2012 Aug 31;345:e5895.
    12. Hout FA, Nienhuis ED, Robben PB, Frederiks BJ, Legemaate J. Supervision by the Dutch Healthcare Inspectorate. Eur J Health Law 2010 Sep;17(4):347-360. [Medline]
    13. Van der Schee E, Groenewegen PP, Friele RD. Public trust in health care: a performance indicator? J Health Organ Manag 2006;20(5):468-476. [CrossRef] [Medline]
    14. Torres E, Vasquez-Parraga AZ, Barra C. The path of patient loyalty and the role of doctor reputation. Health Mark Q 2009;26(3):183-197. [CrossRef] [Medline]
    15. Reichheld FF. The one number you need to grow. Harv Bus Rev 2003 Dec;81(12):46-54, 124. [Medline]
    16. Hekkert KD, Cihangir S, Kleefstra SM, van den Berg B, Kool RB. Patient satisfaction revisited: a multilevel approach. Soc Sci Med 2009 Jul;69(1):68-75. [CrossRef] [Medline]
    17. Daniels JP, Hunc K, Cochrane DD, Carr R, Shaw NT, Taylor A, et al. Identification by families of pediatric adverse events and near misses overlooked by health care providers. CMAJ 2012 Jan 10;184(1):29-34 [FREE Full text] [CrossRef] [Medline]
    18. Vincent C, Davis R. Patients and families as safety experts. CMAJ 2012 Jan 10;184(1):15-16 [FREE Full text] [CrossRef] [Medline]
    19. Emmert M, Sander U, Pisch F. Eight questions about physician-rating websites: a systematic review. J Med Internet Res 2013 Feb 1;15(2):e24 [FREE Full text] [CrossRef] [Medline]
    20. Emmert M, Meier F, Pisch F, Sander U. Physician choice making and characteristics associated with using physician-rating websites: cross-sectional study. J Med Internet Res 2013 Aug 28;15(8):e187 [FREE Full text] [CrossRef] [Medline]
    21. Maramba ID, Davey A, Elliott MN, Roberts M, Roland M, Brown F, et al. Web-based textual analysis of free-text patient experience comments from a survey in primary care. JMIR Med Inform 2015 May 6;3(2):e20 [FREE Full text] [CrossRef] [Medline]


    IGZ: Health Care Inspectorate (Inspectie voor de Gezondheidszorg)
    OR: odds ratio
    PRS: patient rating site

    Edited by G Eysenbach; submitted 19.04.16; peer-reviewed by M Emmert, B Ranard; comments to author 26.05.16; revised version received 09.06.16; accepted 24.06.16; published 15.07.16

    ©Rudolf Bertijn Kool, Sophia Martine Kleefstra, Ine Borghans, Femke Atsma, Tom H van de Belt. Originally published in the Journal of Medical Internet Research (, 15.07.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.