This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Patients are increasingly rating their family physicians on the Internet in the same way as they might rate a hotel on TripAdvisor or a seller on eBay, despite physicians’ concerns about this process.
This study aims to examine the usage of NHS Choices, a government website that encourages patients to rate the quality of family practices in England, and associations between web-based patient ratings and conventional measures of patient experience and clinical quality in primary care.
We obtained all (16,952) ratings of family practices posted on NHS Choices between October 2009 and December 2010. We examined associations between patient ratings and family practice and population characteristics. Associations between ratings and survey measures of patient experience and clinical outcomes were examined.
61% of the 8089 family practices in England were rated, and 69% of ratings would recommend their family practice. Practices serving younger, less deprived, and more densely populated areas were more likely to be rated. There were moderate associations with survey measures of patient experience (Spearman ρ 0.37−0.48,
The frequency of patients rating their family physicians on the Internet is variable in England, but the ratings are generally positive and are moderately associated with other measures of patient experience and weakly associated with clinical quality. Although potentially flawed, patient ratings on the Internet may provide an opportunity for organizational learning and, as it becomes more common, another lens to look at the quality of primary care.
Consumers are using the Internet to rate services and products, for example when they stay in a hotel or buy a product online. This increasingly applies to health care, particularly in the US and UK. A number of websites allowing patients to rate their care have been developed by health care payers and the commercial sector, such as RateMDs or Angie’s List, in an effort to increase transparency and responsiveness of health systems and to help patients choose between providers [
The value of patient rating websites has been criticized, particularly by family physicians [
NHS Choices is a government-run website that serves as a comprehensive directory of health care providers in England and includes comparative information on performance of family practices [
Little is known about who uses patient-rating websites and how unsolicited ratings by patients relate to more conventional measures of patient experience and clinical quality. While some patients may be interested in reviewing other patients’ ratings as a window into family physicians’ interpersonal styles and office amenities, the amount of agreement between these ratings and conventional measures of quality such as patient experience captured in surveys or clinical process and outcome measures will be important in understanding the usefulness of these ratings in quality measurement.
This paper seeks to examine usage patterns of patients’ ratings of family physicians on the Internet. We describe how frequency and nature of rating vary with practice and population characteristics, and we present comparisons between web-based ratings and conventional measures of patient experience and clinical quality.
All patient ratings of family practices posted on the NHS Choices website between October 14, 2009 (the date the function started), and December 31, 2010, were obtained from the Department of Health, aggregated to the practice level. The NHS Choices website allows patients to indicate whether they would recommend a family practice to a friend (yes/no) and rate practices on a scale for four specific domains of quality: whether they were able to get through to the practice by telephone; whether they were involved in decisions about care; whether they were able to get an appointment when they wanted one; and whether they were treated with dignity and respect by staff. Data on the individual characteristics of those leaving ratings online were not available.
Survey measures of patient experience were obtained from the national General Practice Patient Survey at the practice level. This is a large mail-based survey sent to 5.7 million patients in 2009/10, with more than 2.1 million responses received (a response rate of 37%) [
Population and practice characteristics were obtained from the NHS Information Centre. Practice population variables were: the proportion of patients over 65 years, the index of multiple deprivation (IMD) score (an area-based measure of socioeconomic status including components of income, employment, health, education, crime, and housing), the proportion of the practice population who reported their ethnicity as “white”, the population density of the practice (measured as people per square kilometre), and the practice list size. Variables describing the practice were: whether it was a singlehanded practice, a training practice, and the type of contract it had with the health care payer. The type of contract was listed as either Personal Medical Services (PMS) or General Medical Services (GMS). The GMS contract is nationally agreed, while PMS is locally agreed.
We obtained data on clinical quality of family practices from the NHS Information Centre [
In order to create a complete set of family practices in England, we took the NHS’s list of 8381 practices [
In order to link data about the practices from NHS Choices with other practice performance and demographic data, we matched practices by postcode using Excel. We created a computer program that extracted postcodes from the NHS Choices website using Python programming language. This match was checked manually by one person to ensure appropriate linking. Where more than one practice is located at the same postcode, they were manually checked to ensure the correct practice was listed using the names of the practice physicians.
We compared the NHS Choices rating data against patient survey and outcomes data from the 2009/10 financial year. We obtained only incomplete practice demographic descriptive data for 163 practices, but these practices were still included in the analysis as we still had information from the patient survey and NHS Choices. Excluded from this analysis were 24 GP practices (of 5362 with ratings) on NHS Choices that could not be matched to the official list of practices.
We examined associations between whether a family practice was rated and the practice and population characteristics using logistic multivariate regression. A least square regression analysis was used to examine associations between the proportion of patients willing to recommend the practice and practice and population characteristics.
In order to compare ratings with survey measures of patient experience, we compared ratings for questions on NHS Choices with results of the national General Practice Patient Survey using Spearman rank coefficient of correlation. We selected questions from the national survey that most closely matched the ratings questions on the NHS Choice website. In order to compare ratings with measures of clinical quality, we compared the proportion of people who would recommend the practice on NHS Choices with the traditional quality indicators described above using Spearman rank coefficient of correlation. For ratings on a scale, the mean rating for each practice was calculated. Analysis was done with Stata 11 software.
The level of NHS Choices usage to rate family practices online was mapped by ArcMap 9.3 software, using an Inverse Distance Weighted algorithm. The location of each practice was geocoded [
Of the 8089 practices included in the study, 4950 (61%) had been rated on the NHS Choices website. There were 16,952 ratings of these family practices. The mean number of ratings of each practice was 2.1; the median was 1. The range was from 0 to 149 ratings. A histogram of frequency of rating is shown in
Histogram of rating frequency for all GP practices.
Mean patient ratings posted for questions on NHS Choices.
|
|
|
I would recommend this GP practice to a friend. | 64.0 % | 33.3–100% |
I am able to get through to the practice by telephone. | 4.2 out of 5 |
3.7–5.0 |
This GP practice involves me in decisions about my care and treatment. | 4.1 out of 5 |
3.5–5.0 |
I am able to get an appointment when I want one. | 3.6 out of 5 |
3.0–4.3 |
I am treated with dignity and respect by the staff. | 4.1 out of 5 |
3.5–5.0 |
The logistic regression model showed that a larger practice size, a lower proportion of older patients, lower deprivation, higher population density, and not being a singlehanded practice were all positively associated with the likelihood of a practice being rated (
Associations between whether a practice is rated with population and practice characteristics.
|
|
|
|
|
Practice population size |
7587 | 5554 | 15.38 | <.001 |
IMD score of patients |
25.1 | 28.2 | -7.82 | <.001 |
Population density |
458 | 403 | 6.72 | <.001 |
Singlehander |
10 | 20.7 | -4.50 | <.001 |
Percentage of population aged over 65 years | 15.1 | 15.6 | -3.88 | <.001 |
Percentage of population who are white | 87.3 | 88.3 | -1.58 | .11 |
Type of contract |
42.7 | 39.6 | -0.71 | .48 |
Training practice |
32.3 | 22.0 | 0.35 | .73 |
The least square regression model showed that smaller practice size, a higher proportion of white patients, lower population density, lower deprivation, being a training practice, and not being a singlehanded practice were all positively associated with higher levels of recommendation (
Associations between the proportion of patients recommending a practice with population and practice characteristics.
|
|
|
Proportion of population who are white | 7.39 | <.001 |
Training practice | 6.61 | <.001 |
Practice population size | -4.61 | <.001 |
Population density | -4.17 | <.001 |
IMD score of patients | -3.92 | <.001 |
Singlehander | -3.76 | <.001 |
Proportion of population aged over 65 years | 1.78 | .075 |
Type of contract | 0.62 | .53 |
Associations between patient ratings and patient experience measures from the family practice survey had Spearman ρ values of between 0.36 and 0.48 and were all significant at the
Associations between web-based patient ratings and conventional survey measures of patient experience.
|
|
|
|
I would recommend this GP practice to a friend. | Recommending GP practice to someone who has moved to the local area—% yes | 0.48 | <.001 |
I am able to get through to the practice by telephone. | Ease of getting through on the phone—% easy | 0.43 | <.001 |
This GP practice involves me in decisions about my care and treatment. | Rating of doctor involving you in decisions about your care—% good | 0.38 | <.001 |
I am able to get an appointment when I want one. | Able to book ahead for an appointment with a doctor in the past 6 months—% yes | 0.37 | <.001 |
I am treated with dignity and respect by the staff. | Rating of doctor treating you with care and concern—Good | 0.39 | <.001 |
Associations between web-based patient ratings of whether the user would recommend the GP practice to a friend and various clinical quality indicators.
|
|
|
Proportion of patients with diabetes receiving flu vaccination | 0.07 | <.001 |
Controlled blood pressure in hypertensive patients (systolic/diastolic less than 150/90 mm Hg) | 0.07 | <.001 |
Controlled HbA1C in patients with diabetes (less than 7%) | 0.06 | <.001 |
% low-cost statin prescribing | -0.03 | .02 |
Cervical screening rate | 0.18 | <.001 |
Admission rates for ambulatory care sensitive conditions | -0.15 | <.001 |
Total clinical QOF points available | 0.11 | <.001 |
Map of frequency of primary care ratings on the NHS Choices website.
The positive nature of responses (with 64% recommending) on the NHS Choices website is in line with other studies of patient-reported Internet feedback in acute hospitals in the UK (where 68% recommend) [
The results demonstrate that usage of patient ratings of family practices via the NHS Choices website has been variable with some practices having more than 100 ratings, but many having none, and an average of only two ratings per practice over the 15-month period covered. A sizable minority (39%) of practices never received a single rating. The data analyzed here represent the initial period of the rating function being available, so usage may well increase as it becomes more popular. However, this usage of online ratings is higher than noted in other settings. A study in Boston, MA, found only 81 of a random sample of 300 physicians had been rated online [
The results suggest that the level of usage of online ratings is different in different communities. Practices serving younger, more urban, and less deprived communities were more likely to be rated. This is in line with previous work showing more usage of the Internet as a health resource in those groups [
The demonstration of an association between practice population characteristics and rating usage does not prove a link between individual characteristics and usage; there is a risk of the ecological fallacy, in which an incorrect inference can be drawn about individuals based on aggregated statistics about a group of people. However, these results do confirm that usage rates are variable around the country and suggest that individual characteristics may have a role in influencing usage. Further studies, using individual level data, are required to understand the characteristics of those using ratings websites. The results also show that different practice characteristics are associated with different levels of satisfaction with service, measured as willingness to recommend. This is in line with a wide body of literature on patient experiences, which notes that ethnicity and different socioeconomic factors are associated with satisfaction with medical care [
Moderate associations between patients’ ratings left on the web and more conventional surveys of patient experiences were found (Spearman ρ values of between 0.38 and 0.48). Due to the large number of practices in the analysis, these associations are all highly significant.
The association between ratings and clinical outcomes is less convincing. Although the results do show increasing levels of recommendation associated with better care across a number of indicators, for many of them the strength of the association is weak and significant only because of the large number of data points. This may be because there is a genuine tension between patient experience and some aspects of technical quality; it is possible to do all of the right things technically in health care and still get a bad outcome. It is also possible that patients’ personal values may differ from the public health perspective captured in the quality metrics. Both the associations with clinical outcomes and survey measures of experience are consistent with similar findings at the hospital level [
These data suggest that we can be more confident in the use of online rating data as a measure of patient experience, despite the many fundamental biases that are an inevitable consequence of this sort of data. However, the extent to which online ratings reflect the technical quality of clinical primary care is less clear. As momentum develops around the need to capture the patient’s experience, these ratings represent a potentially valuable source of information about quality of care when taken with other more conventional measures. Therefore, as such ratings become more common, systems should be developed to allow patients to examine these data alongside other more traditional outcome data, presented in a digestible and accessible way. Similarly, family physicians and practices should develop strategies to respond to these comments in constructive ways [
Our study has a number of limitations. We removed practices with a size below 1000 patients, as has been done in other analyses of practice performance [
Our study also has strengths. It uses a novel, largely complete national set of online ratings. The unique nature of the NHS’s performance related pay system provides detailed information on clinical performance at the practice level, allowing detailed comparisons to be made.
These findings represent the first national review of an online ratings system in primary care. Our findings lend support to some of the arguments against online rating systems, particularly that they may have a selection bias. However, they also suggest some positive aspects. They are not just used as a mechanism to complain, as more often than not they are positive, and there are moderate associations with more traditional measures of patient satisfaction. As the numbers of rating per practice are low, they may be less useful as a measure of quality in family practice but they may provide a novel route for organizational learning. However, as numbers of ratings rise, it is likely that they will become a more useful tool for patients to make informed choices about where to receive their care.
The Department of Primary Care & Public Health at Imperial College London is grateful for support from the NIHR Collaboration for Leadership in Applied Health Research & Care (CLAHRC) Scheme, the NIHR Biomedical Research Centre scheme, and the Imperial Centre for Patient Safety and Service Quality. CM is funded by the Higher Education Funding Council for England and the National Institute for Health Research.
We are grateful to the staff at NHS Choices for providing the data for this analysis, in particular John Robinson.
Professor Darzi was Parliamentary Under-Secretary of State (Lords) in the United Kingdom Department of Health from 2007 to 2009. Professor Majeed has received funding for research from Dr Foster Intelligence, a commercial health intelligence company. Professor Majeed is also a General Practitioner in the NHS general practice of Dr Curran & Partners, which has a page on NHS Choices. For the other authors, no conflicts are declared.