This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Many online physician-rating sites provide patients with information about physicians and allow patients to rate physicians. Understanding what information is available is important given that patients may use this information to choose a physician.
The goals of this study were to (1) determine the most frequently visited physician-rating websites with user-generated content, (2) evaluate the available information on these websites, and (3) analyze 4999 individual online ratings of physicians.
On October 1, 2010, using Google Trends we identified the 10 most frequently visited online physician-rating sites with user-generated content. We then studied each site to evaluate the available information (eg, board certification, years in practice), the types of rating scales (eg, 1–5, 1–4, 1–100), and dimensions of care (eg, recommend to a friend, waiting room time) used to rate physicians. We analyzed data from 4999 selected physician ratings without identifiers to assess how physicians are rated online.
The 10 most commonly visited websites with user-generated content were HealthGrades.com, Vitals.com, Yelp.com, YP.com, RevolutionHealth.com, RateMD.com, Angieslist.com, Checkbook.org, Kudzu.com, and ZocDoc.com. A total of 35 different dimensions of care were rated by patients in the websites, with a median of 4.5 (mean 4.9, SD 2.8, range 1–9) questions per site. Depending on the scale used for each physician-rating website, the average rating was 77 out of 100 for sites using a 100-point scale (SD 11, median 76, range 33–100), 3.84 out of 5 (77%) for sites using a 5-point scale (SD 0.98, median 4, range 1–5), and 3.1 out of 4 (78%) for sites using a 4-point scale (SD 0.72, median 3, range 1–4). The percentage of reviews rated ≥75 on a 100-point scale was 61.5% (246/400), ≥4 on a 5-point scale was 57.74% (2078/3599), and ≥3 on a 4-point scale was 74.0% (740/1000). The patient’s single overall rating of the physician correlated with the other dimensions of care that were rated by patients for the same physician (Pearson correlation,
Most patients give physicians a favorable rating on online physician-rating sites. A single overall rating to evaluate physicians may be sufficient to assess a patient’s opinion of the physician. The optimal content and rating method that is useful to patients when visiting online physician-rating sites deserves further study. Conducting a qualitative analysis to compare the quantitative ratings would help validate the rating instruments used to evaluate physicians.
In 2010, 88% of adult Americans used the Internet to search for health-related information [
Many physician-rating websites provide users with basic information about the physician such as years in practice and contact information [
Many websites enable users to enter reviews and rankings about specific physicians. This capability has drawn the attention of consumer advocacy groups, providers, insurance companies, and hospitals. Although knowledge about the patient experience is useful, critics of these portals identify them as being at risk for misinformation, sabotage, and manipulation [
The goals of this study were to (1) determine the most frequently visited physician-rating websites that have user-generated content, (2) evaluate the content characteristics of each site to rate physicians, and (3) analyze online ratings of 4999 individual physician ratings.
Approval for this study was obtained from the Institutional Review Board at Stanford University School of Medicine.
A search of the Internet (Bing, Google, Google Directory, Google Trends, Blekko, Yahoo, and Yahoo Directory) with search terms
We then studied each site to determine the types of rating scales (eg, 1–5, 1–4, 1–100) used and dimensions of care rated (eg, recommend to a friend, waiting room time). All the dimensions of care were identified for each website. To compare different websites, we created a semantic normalization tool. A semantic conversion table was created by first identifying all the different dimensions of care used on each website (
Semantic conversion table used to normalize different dimensions of care used to rate physicians on the websites
Overall rating | Communication Skills | Access | Facilities | Staff |
Overall | Communication | Appointments | Office cleanliness | Courteous staff |
Level of trust | Explanation | Approachable | Office setting | Staff |
Overall quality of care | Explanation of medications | Doctor availability | Office environment | Staff friendliness |
Recommendation | Follow-up | Convenience | Service | Staff helpfulness |
Recommend to a friend | Attentive during visit | Ease of appointment | Waiting room | Staff professionalism |
Patient satisfaction | Listens and answers questions | Quality of referrals | Facilities | Office friendliness |
Likely to recommend | Bedside manner | Make Referrals | ||
Helps patient understand | Punctuality |
Raw data without specific physician identifiers were obtained in October, November, and December 2010 via a nonrandom selection of 4999 online physician ratings from 23 multiple specialties (allergy, cardiology, cardiothoracic surgery, dermatology, endocrinology, gastroenterology, general surgery, hematology, internal medicine, nephrology, neurology, neurosurgery, obstetrics and gynecology, oncology, ophthalmology, orthopedic surgery, otolaryngology, pediatrics, plastic surgery, primary care, pulmonary medicine, rheumatology, and urology) in 25 metropolitan areas (Atlanta, GA; Austin, TX; Baltimore, MD; Boston, MA; Charlotte, NC; Chicago, IL; Colorado Springs, CO; Columbus, OH; Denver, CO; Houston, TX; Los Angeles, CA; Miami, FL; Minneapolis, MN; New Orleans, LA; New York City, NY; Orlando, FL; Phoenix, AZ; Portland, OR; Salt Lake City, UT; San Diego, CA; San Francisco, CA; Raleigh, NC; San Jose, CA; Seattle, WA; and Washington, DC). We chose these cities because they have the highest Internet usage and largest population in the United States [
The number of reviews collected from each website varied proportionally by how frequently the websites were visited based on Web traffic estimates from Google Trends. Therefore, the number of reviews from each website was proportional to Web traffic volume assuming that search patterns on Google are similar to those on other search engines.
The sequence of steps followed to acquire each physician rating was to visit the website, enter the city, choose a specialty, enter the largest search radius, and then sort physicians by name when possible. If sorting by name was not possible then location was used. Only reviews that had at least one physician rating completed by a patient within the years 2000–2010 were included in the analyses. Each analyst was assigned a set of metropolitan areas to evaluate physician data.
Cut-offs of 75 (100-point scale), 4 (5-point scale), and 3 (4-point scale) were used to define the favorable threshold for each category of physician-rating website. To compare rankings from different websites with the same rating system, we used a weighted average to accurately represent the overall compiled rating. Only physician-rating sites with the same rating system were compared with one another.
To facilitate analyses, similar dimensions of care—but with different terms used by each website—were grouped into 1 of the 5 categories defined above (
The 10 most commonly visited online physician-rating websites with user-generated content per Google Trends were HealthGrades.com, Vitals.com, Yelp.com, YP.com, RevolutionHealth.com, RateMD.com, Angieslist.com, Checkbook.org, Kudzu.com, and ZocDoc.com (
Top 10 most frequently visited physician-rating websites as a relative measure of Web traffic as measured through Google Trends (October-December 2010)
Website | Percentage | Daily unique visits (per Google Trends) |
HealthGrades | 40% | 254,600 |
Vitals | 20% | 127,300 |
Yelp | 15% | 95,475 |
Checkbook | 7% | 44,555 |
YP | 5% | 31,825 |
ZocDoc | 4.8% | 30,552 |
AngiesList | 3.2% | 20,368 |
RateMD | 3% | 19,095 |
RevolutionHealth | 1% | 6365 |
Kudzu | 1% | 6365 |
Total | 100% | 636,500 |
Patients rated 35 different dimensions of care in the websites, with a median of 4.5 (mean 4.9, SD 2.8, range 1–9) dimensions of care per website (
Information available on the top 10 physician-rating sites
Website | Comments | Board certification | Years in practice | Physician comparison | Advertising | Sanctions |
RateMD | Yes | No | Yes | No | Yes | No |
Vitals | Yes | Yes | Yes | Yes | Yes | Yes |
AngiesList | Yes | No | Yes | No | No | No |
HealthGrades | No | Yes | Yes | No | Yes | Yes |
YP | Yes | No | No | No | Yes | No |
Kudzu | Yes | No | No | No | Yes | No |
Yelp | Yes | No | No | No | Yes | No |
ZocDoc | Yes | Yes | No | No | No | No |
CheckBook | No | Yes | Yes | Yes | No | No |
RevolutionHealth | Yes | Yes | Yes | No | Yes | No |
The average rating was 77 (308/400, 77.0%) for sites using a 100-point scale (SD 11, median 76, range 33–100). For sites using a 5-point scale the average rating was 3.84 (76.8%, 2764/3599, SD 0.98, median 4, range 1–5). For sites using a 4-point scale the average was 3.1 (77.5%, 774/1000, SD 0.72, median 3, range 1–4).
The percentage of reviews with a rating of 75 or higher on physician-rating sites with a 100-point scale was 61.5% (246/400). The percentage of reviews with a rating of 4 or higher on sites with a 5-point scale were 57.74% (2078/3599). The percentage of reviews with a rating of 3 or higher on sites with a 4-point scale were 74.0% (740/100) (
Physician ratings from the top 10 physician-rating websites with user-generated content. Percentage favorable ratings defined as ≥3 of 4, ≥4 of 5, or ≥75 of 100
Website | Number of |
Percentage |
Favorable reviews | Overall rating | Lowest |
Highest |
||||
n | % | Mean | SD | Median | ||||||
|
||||||||||
Checkbook.org/PatientCentral | 350 | 7% | 217 | 62 | 77.59 | 10.48 | 76.00 | 34.00 | 100.00 | |
RevolutionHealth | 50 | 1% | 29 | 57 | 74.24 | 16.01 | 76.00 | 33.00 | 100.00 | |
Weighted average | 400 | 8% | 246 | 62 | 77.17 | 11.17 | 76.00 | 33.00 | 100.00 | |
|
||||||||||
AngiesList | 159 | 3% | 103 | 65 | 3.95 | 0.95 | 4.00 | 1.00 | 5.00 | |
HealthGrades | 2000 | 40% | 1139 | 57 | 3.82 | 0.98 | 4.00 | 1.00 | 5.00 | |
Kudzu | 49 | 1% | 26 | 53 | 3.74 | 0.96 | 4.00 | 1.00 | 5.00 | |
RateMD | 150 | 3% | 87 | 58 | 3.84 | 1.00 | 4.00 | 1.00 | 5.00 | |
Yelp | 750 | 15% | 442 | 59 | 3.86 | 0.97 | 4.00 | 1.00 | 5.00 | |
YP | 250 | 5% | 158 | 63 | 3.93 | 0.92 | 4.00 | 1.00 | 5.00 | |
ZocDoc | 241 | 5% | 123 | 51 | 3.77 | 0.92 | 4.00 | 1.00 | 5.00 | |
Weighted average | 3599 | 72% | 2078 | 58 | 3.84 | 0.98 | 4.00 | 1.00 | 5.00 | |
|
||||||||||
Vitals | 1000 | 20% | 740 | 74 | 3.10 | 0.72 | 3.00 | 1.00 | 4.00 | |
Total | 4999 | 100% | 3064 | 61.28 |
The multiple dimensions of care rated by patients on the physician-rating sites with a 5-point scale had a strong correlation with the overall rating (Pearson correlation,
Distribution of percentile ratings for each dimension of care rated on all physician-review sites.
Pearson correlation comparing overall rating versus staff rating (n = 4999, Pearson correlation,
This analysis of 4999 physician ratings across 10 websites revealed that approximately 2 out of 3 patient reviews are favorable. These results are consistent with a study that found that 88% of 190 reviews of 81 Boston physicians were favorable [
In all, 35 different dimensions of care were rated by patients in the websites, with an average of 5 questions per site. There was a high correlation between the overall rating of the physician and the other dimensions of care rated (access, communication skills, facility, and staff). This is consistent with using net promoter score methodology to measure customer satisfaction [
Many physicians will take the position that online review sites do not give insight into quality of care. This is valid since obtaining consensus on the definition of quality, even among experts, is challenging. However, patient satisfaction ratings and comments do offer insight into a patient’s experience. As more user-generated content is added, the value of ratings will increase. Patient satisfaction is derived from several factors including the baseline expectation of the patient [
Despite the overall favorable rating of physicians by patients, the topic of physician ratings is rather sensitive [
Physician-rating websites hosted by insurance companies have been questioned because of the conflict of interest that insurance companies have by reporting data that can potentially drive patients to providers that are cheap and not because they are good [
This study has several limitations. There is an implicit selection bias to websites that depend on the user to actively engage the review site and write a review. In the future, to get more feedback, providers may bundle review requests with online services such as appointments (eg, ZocDoc.com) and social networking sites. This may reduce the selection bias that limits the value of physician ratings. We derived physician-rating site traffic from Google Trends, which is not an absolute measure of total site traffic. Also, the authenticity of the review may be in question [
None declared
Video of the presentation of Dr Kadry at the Medicine 2.0 Congress at Stanford University, September 18th, 2011