The initial search identified 1628 articles. After eliminating duplicates and a review of titles and abstracts, 260 studies remained for detailed reflection (see Figure 1). Screening of reference lists, expert consultation, and Internet searches yielded 22 additional articles. Finally, 24 articles met our inclusion criteria. It is worth mentioning that the papers included vary considerably by inclusion criteria and focus. The result is a wide range in the number of included studies. Furthermore, all studies were published either in English or German. We did not find a study containing an English language abstract in another language (eg, Spanish, French) during our review process. Most papers have been published by US (n=13) or German authors (n=8). Two studies were published in 2007, five studies in 2009, eight in 2010, four in 2011, and five in 2012.

First, we investigated the number of ratings on physician-rating websites. One study estimated the number of ratings on physician-rating websites from a national perspective. Data for US physicians obtained from RateMDs showed that 16% (112,000 out of approx. 700,000) of national physicians were assessed by January 2010 [7]. Some studies regarded a sample of physicians to conduct analyses. Lagu and colleagues showed that 81 out of 300 Boston physicians had been rated (27%) [2]. In another study, Mostaghimi and colleagues counted that, out of 250 randomly selected internal medicine physicians, 53 physicians (21%) were rated on Healthgrades, 13 (5%) on RateMDs, and 1 physician on Wellness (0.4%), respectively. Most of the 250 physicians had still not been rated (Healthgrades: 69%, RateMDs: 61%, Wellness: 98%, respectively) [23]. In 2009, the percentage of rated physicians for ten different specialty/region combinations on five German physician-rating websites varied between 0% (eg, for urologists in Frankfurt) and 100% (radiologists in Hannover). The overall mean percentage of rated physicians was reported to be at between 3.36% (Patienten-empfehlen-Ärzte) and 25.78% (Medführer), respectively [21]. In a recently published study, between 3% and 28% of a random sample of physicians had been rated at least once [24].

Regarding the average number of ratings per physician on physician-rating websites, results for the American physician-rating website RateMDs were reported to be 2.7 mean ratings (range = 1-103) in 2009 [25] and 3.2 in January 2010 [7]. Nearly half of the physicians had only a single rating on RateMDs in 2010, and the number of physicians with five or more ratings was 12.5% [7]. For a sample of 300 Boston physicians, 190 reviews had been posted in total, ie, the mean number of ratings was 0.63 per physician. If only rated physicians (n=81) were analyzed, the mean number increased to 2.35 ratings [2]. Regarding a subsample of 250 randomly selected physicians in Boston, between one and four reviews could be found with 29% on Healthgrades, 39% on RateMDs, and 2% on Wellness, respectively. On Healthgrades, only 3 physicians (2%) had more than five reviews; no physician with five or more ratings could be found on RateMDs or Wellness [23]. German results from 2010 show similar findings; the number of ratings for physician-rating websites was reported to be 600,000 for Jameda, 450,000 for DocInsider, 150,000 for Arztauskunft, and 73,000 for Imedo, respectively. Compared to the total number of physicians in the German outpatient sector (approximately 150,000), the highest mean of approximately four evaluations per physician could be tracked on the website Jameda [6]. Another study determined a mean number of ratings of between 1.1 and 3.9; the maximum number of ratings per physician varied across the physician-rating websites at between 3 and 27 [24].

There is little evidence available to answer the question of whether there are differences among rated physicians according to their socioeconomic status. Only three studies provided evidence relevant to this question. One previously published study showed that 74% of rated physicians were male (national average 72%). It was also shown that physicians who were board certified, and those who had at least one paid malpractice claim, were more likely to be rated. The authors could further show that younger physicians were much less likely to be rated. Graduates of more highly ranked medical schools and those of lower-ranked medical schools were rated with nearly the same frequency [7].

With respect to subgroups of physicians, Lagu and colleagues determined the number of rated generalists (37) and subspecialists (38) to be quite similar [2]. In absolute terms, primary care physicians were more likely to be rated than other specialties such as surgeons or obstetrics/gynecologist [7]. Another study showed that nonsurgical subspecialties, as well as OBGYN (Obstetrics Gynecology) & IVF (in-vitro fertilization), were most rated (22% and 19%, respectively). The lowest percentage of ratings was determined for doctors specialized in physical medicine and general surgery (1% and 2%, respectively) [25]. In relative terms, compared with the national physician composition, rated physicians were most likely to be obstetrician/gynecologists (32%). The likelihood of being rated for other specialties was calculated as follows: 25% of medical specialists, 20% of surgeons, 16% of primary care physicians, and 7% of physicians classified as other specialists (such as radiologists, pathologists, and anesthesiologists) [7]. Black et al showed that nonsurgical subspecialty and internal medicine physicians have been rated most (28% and 13%, respectively). In contrast, general surgery and physical medicine physicians have been rated the least (3% and 1%, respectively) [25]. Also, the numbers of individual ratings varied across specialty; the highest numbers were counted for OBGYN and IVF, dermatology, and cosmetic surgery (mean 4.4 individual ratings) and the lowest for pediatrics and general surgery (mean 1.8 ratings) [25].

In total, six studies provide information on the results of the ratings. Two studies focused on the US site RateMDs and found the overall reviews to be quite positive. On a scale of 1 to 5, the mean score was reported to be 3.93 [7] and 3.82 [25], respectively. A comprehensive analysis of German physician-rating websites confirmed that most ratings were positive. Here, the mean rating was between 1.1 and 1.5 (3-point scale, 1 “good”, 3 “poor”) [24]. When assessing the 10 most commonly visited US physician-rating websites, the aggregated mean ratings were as follows [22]: 77 out of 100 when using a 100-point scale (SD 11), 3.84 out of 5 (77%) for sites using a 5-point scale (SD 0.98), and 3.1 out of 4 (78%) for sites using a 4-point scale (SD 0.72). It was further reported that the percentage of reviews rated ≥75 on a 100-point scale was 61.5%, ≥4 on a 5-point scale was 57.74%, and ≥3 on a 4-point scale was 74.0% [22]. On RateMDs, 45.80% of the physicians received the best score and only 12% were rated with the worst score [7]. Other studies do not provide a mean rating but give further information about the percentage of positive and negative reviews. Lagu und colleagues did the same when they reported that the vast majority (88%) of reviews were positive, only six percent were negative, and six percent were neutral [2]. On Canadian RateMDs, 70% of the comments were reported to be favorable and about 30% comments were negative [26].

Exploring in more detail, the rating differences between physician groups was assessed in some studies. Gao et al found similar mean ratings for physicians in primary care (4.02), medical specialties (3.96), surgeon and surgical specialties (3.89), and obstetrician/gynecologists physicians (4.01). They further demonstrated that physicians listed within the group of other specialties had lower ratings (3.59) [7]. Others reported the highest mean scores for pediatricians, general surgery, and subspecialty surgery (4.22, 4.10, and 4.07, respectively) [25]. Lagu et al determined that generalists and subspecialists had a similar percentage of positive, negative, and neutral reviews [2]. Furthermore, male physicians, younger physicians, board-certified physicians, and those graduating from a top-50 medical school were shown to have statistically significant better ratings [7].

Patients have the choice of writing narrative commentaries in free text form on 86% of English-language and German-language physician-rating websites [14]. Physicians’ critiques often concern these narratives, as they might provide the opportunity for doctor-bashing, defamation, etc. However, obtaining actionable information might help physicians to change communication style, facility, or staff. Such information may be better obtained by those narratives, rather than by a scaled survey displaying numbers or stars. A single quantitative rating of 1 out of 5 stars does not provide further assistance for improvement. But, if comments show that the exam rooms were dirty, then the provider will better understand the low rating [22].

Numbers on how many physicians have been rated by means of a patient narrative are quite scarce. According to US evidence, there is at least one narrative rating for approximately 17% of physicians [2]. Alemi and colleagues showed that the mean number of patient narratives per physician for a sample of 200 rated physicians by means of a patient narrative was 9 (range from 1 to 57, SD 8.10) [27]. Furthermore, the authors found that narratives were mostly positive (89%) [2]. In another study, Lopez et al qualitatively analyzed 712 narratives for internists and family practitioners from RateMDs and Yelp and found that 63% of the narratives contained positive comments [8]. In an analysis of 995 narratives from RateMDs, it could be determined that 69% (688) were praise, 21% (210) were complaints, and 10% (97) were both [27]. This result was confirmed by Black et al, showing that positive terms (54.1%) were more frequent than negative terms (16.0%) [25]. Thereby, the five most common positive terms were good, knowledgeable, best, excellent, and wonderful. In contrast, the most common negative terminology found was rude, bad, worst, horrible, and terrible [25]. The mean length of the narratives on RateMDs was 19.3 words [25]. Finally, Alemi and colleagues coded narratives with several reasons for dissatisfaction into nine categories, such as (1) physician-related concerns, (2) staff-related issues, (3) getting in to be seen, etc. As a result, most comments were related to aspects of category 1 (eg, doctor’s advice and treatment, time doctor takes, explanations provided by the doctor) and category 2 [27].

Many physicians are uncertain about how to deal with physician-rating websites. Authors seem to agree that physician-rating websites will play a major role in health care in the future, and physicians should therefore not underestimate, but instead recognize, the popularity of such websites [23,25,28,29]. Physicians should be further aware of the fact that not only patients, but also insurance companies or even other physicians check these sites; the latter, for instance, to get information about job candidates [30].

Physicians should perform “self-audits” on popular physician-rating websites to search for available information [23,30], like their scores [29]. Therefore, it may be helpful if a staff member monitors these sites on a regular basis. If nothing else, physician-rating websites often provide incorrect demographic information (eg, incorrect address, links to old practices, opening hours), which should be corrected [23,30]. Next, physicians should use the ratings in order to evaluate their patients’ satisfaction [31]. Since existing measures of patient experiences do not seem to facilitate a good understanding for health care providers, personalized feedback on physician-rating websites may be advantageous. Rather than departmental reports or annual surveys, anonymous Internet-based reviews may help health care providers improve the quality of care [25]. Furthermore, measures such as medical training rarely give the opportunity to hear what patients want or value because in the real workplace, disappointed patients rarely tell doctors their true opinions. So, patients’ true thoughts on what makes a good doctor, what they value, etc., can be understood [29].

In the case of negative reviews, it is best not to respond online to try to refute the negative review point by point [30]. Further, negative reviews may help providers to create a more patient-centered office environment. Negative interpersonal reviews underscore the importance of a well-perceived bedside manner for a successful patient-physician interaction. Staff, access, etc., affect patient´s reviews as well [8]. However, the challenges of finding a remedy for negative ratings are daunting. An alternative option is to treat the problem before it becomes a problem. The legal company Medical Justice offered to provide doctors with a contract for treatment that includes a clause requiring patients to ask their doctor’s permission before posting a review to a website [30,32,33] (it is worth mentioning that the company has since stopped that practice). Another approach is to politely encourage satisfied patients to submit their own reviews on the most popular physician-rating websites [26,30]. Additionally, positive comments from patients should be posted on one’s own website [30]. Finally, physicians should not make a referral decision based upon results on physician-rating websites, as results related to patient satisfaction and outcome measures are not risk-adjusted and therefore cannot be regarded as reliable [6].

Next, we discuss the major shortcomings of physician-rating websites.

Several strategies have been suggested for further improvement of physician-rating websites. These address some of the limitations mentioned above:

Further recommendations advise that inappropriate content must be edited, users must register with an email [37], physician-rating websites should not contain advertising or official messages, as consumers value independence [38], and that ratings must be transmitted to the provider, albeit anonymously [38].

Five papers were identified, and they all concluded that only a small percentage of physicians have been rated so far on a physician-rating website (eg, 16% of US physicians on RateMDs). As a result, the ratings shown are not likely to be representative of average patient experiences or consumers [6,23,29,30,36,37]. However, physician-rating websites have been gaining an increasing number of ratings over the last years (a 100-fold increase in the United States from 2005 to 2010) [7]. One reason for the low usage might be that patients are still unaware of these websites. A representative survey of 2048 German citizens showed that only 10% of respondents had used physician-rating websites in 2011; however, the number in 2010 was only 7% [39].

Regarding the mean number of ratings on physician-rating websites, US results were reported to range between 0.63 [2] and 3.2 ratings per physician [7]. German results ranged between 0.5 and 4 ratings per physician [6,24]. Thereby, most ratings are given for a low percentage of physicians, meaning that most physicians still remain unrated and those rated have a larger number of ratings. A large US study reported that half of the physicians had only a single rating, and the number of physicians with five or more ratings was 12.5% on RateMDs in 2010 [7]. Consequently, the benefit of such sites for patients still remains limited because more physicians must be rated. The mean number of ratings has to increase to provide a larger benefit to society. However, this might be solved with an increasing awareness level of rating portals.

In total, three studies provided evidence on this question. Certain factors seem to increase the likelihood of being rated on a physician-rating website, such as being older, being male, being board certified, and having at least one paid malpractice claim. Furthermore, some specialties such as primary care physicians and obstetrician/gynecologists seem to influence the likelihood of being rated [7]. Specifically, this includes physicians who have more direct patient contact or those who treat population groups who are more likely to use the Internet actively, such as a younger and female patient population [7].

Some authors expressed concerns whether physician-rating websites might become a channel for disgruntled patients [7]. However, this cannot be confirmed, since international results showed that most ratings express a positive opinion about physicians. One US study determined 88% positive, 6% negative, and 6% neutral ratings [2]. A comprehensive US study confirmed this by showing the mean ratings according to different scoring scales [22]. In sum, studies confirmed that most reviews are on the extreme end, meaning either positive or negative. The studies suggest that most ratings are positive and therefore that some physicians’ concerns may be exaggerated [37].

Our results show that, up to this point, a low number of physicians have been rated by means of a patient narrative; one US study reported a number of 17% [2]. Furthermore, most opinions in narratives are positive (numbers range between 63% and 89%, respectively) [2,8,27]. Physicians’ concerns are about “doctor-bashing”, defamation, etc. However, no evidence has been found to sustain this concern. Adequate measures seem to be in place before comments are published. German physician-rating websites were reported to have implemented adequate word filters, manual provider review, etc. [21], which seem to be effective. Thus, the risk of defamation of physicians in patient narratives seems to be low. Furthermore, it is worth mentioning that physicians also seem to manipulate information on physician-rating websites [2].

According to the literature, physician-rating websites might play a major role in future health care; therefore, physicians should not underestimate, but instead recognize, the popularity of such websites [23,25,28,29]. We showed that getting an overview of the physician-rating websites is recommended, as well as staying on top of the available information on a regular basis. However, no general guidelines are available on how to deal with physician-rating websites. While some may continue to ignore physician-rating websites (due to higher age, little engagement, etc.), others may seek this information in order to be informed, and still others will try to obtain as many positive reviews as possible.

While some flaws are of minor importance, there are also some very major ones. In our estimation, the most important flaw is that physician-rating websites are not able to identify the best physician for a specific intervention or disease. Therefore, the information provided is both too little and not (disease) specific enough. However, it should be debated whether physician-rating websites are really supposed to achieve that. It is more likely that physician-rating websites can give some limited impression of, and only of, patient satisfaction and some structural information. But even these results have to be viewed with caution [6,21,28].

Frequently discussed improvement recommendations relate to the feedback survey. Some argue that a long and detailed survey is necessary to assess the quality of care received. However, the more questions a patient has to answer, the less likely they are to complete the survey [22]. Therefore, a single question is supposed to be sufficient. However, internationally established and validated instruments, such as the Patient Satisfaction Questionnaire from RAND Health (50 items), are more detailed and contain more questions to derive specific results. Consequently, from a researcher’s point of view, the application of validated instruments should be preferred.

It also seems to be the predominant opinion that narrative comments should be integrated on physician-rating websites [22,24]. Of course, this also means that the physician-rating website provider has to establish certain quality measures. However, from the point of view of a patient or physician, the benefits justify it. If physician-rating websites are intended to provide real support to patients, then additional information has to be integrated on the websites, and outcomes must be risk-adjusted [6]. For other sectors of health care (eg, hospitals), risk-adjusted outcome measures are increasingly available. In Germany, the Aqua Institut (www.sqg.de) provides quality indicator data about most German hospitals, and a growing segment of this data is available for PR. In the United States, the Centers for Medicare and Medicaid Services (CMS) offer the website, Hospital Compare. Thus, in the outpatient sector, both outcome information and ratings from patients are available. However, measures about physicians are less available. In Germany, the Aqua Institut has recently started to collect outcome measures about physicians. In the United States, the CMS recently launched Physician Compare, a website publishing data on quality measures for covered professional services provided to Medicare beneficiaries. Consequently, we expect a growing number of risk-adjusted outcome quality indicators to be available in the future for PR about physicians’ quality as well.

Our systematic review has several limitations. It was based on searches in seven databases, and we included articles containing at least an abstract in English. So, it is possible that additional papers exist that were not included. We further concentrated on papers dealing with websites on which individual physicians can be rated. Consequently, knowledge coming from the assessment of websites on which provider organizations or entire hospitals can be rated is not included in our review. Due to the time constraints of our research (up to May 2012), it may be the case that some recently published papers are not included. By focusing only on peer-reviewed literature, we may have missed information in the grey literature that could also have been of interest in attempting to answer some of our questions. Furthermore, due to study heterogeneity, we did not carry out any study appraisal. Before conducting this review, we conducted some interviews with physicians, patients, and physician-rating website providers to get an impression of important questions. However, there may be other relevant questions to discuss, which we did not identify.

To our knowledge, this is the first systematic review of physician-rating websites. Our research shows that the current usage of physician-rating websites, with respect to the number of ratings, is still low but is increasing. Most ratings express positive opinions; this is true for the results of both predetermined rating systems and patient narrative comments. Although negative ratings were mentioned across the different studies, there was no evidence that they are worse via this particular mechanism. Consequently, we could not find any evidence of doctor-bashing in any of the studies. Physicians should not ignore these websites but instead should monitor the available information and use it for internal and external purposes. Several shortcomings limit the significance of the results published on physician-rating websites, and some suggestions on improvement were shown to address them.

The literature suggests several fields for future research. These include investigating how to generate and communicate adequate information for patients. Although a lot of information is provided on physician-rating websites, it remains unclear why patients use these platforms and what the relevant information is about physicians that should be considered for publication on physician-rating websites [28,38]. In this regard, it remains unclear whether patients actually understand the information provided enough to make correct choices [37,38]. Furthermore, research should specifically consider the requirements of disadvantaged people (eg, culture, sex, age, education, socioeconomic group, disability, and health status) to find out whether there are any barriers for certain population groups when seeking and using information provided on physician-rating websites [20,28,38]. The cost-effectiveness of physician-rating websites must be investigated to assess whether [28] the effects of the websites (eg, patient steerage, quality improvement) are large enough to be viewed as money well spent. In this context, the usability of physician-rating websites seems to be crucial. Studies have shown these sites to be neither user-friendly nor patient-centered [2]. Others state that the handling of some physician-rating websites is too complex for some users with respect to the clarity of the physician-rating website or offered search options [34]. Finally, policy makers could contribute to the development of such sites by establishing a regulatory framework to foster the availability of data assessing the quality of care of physicians. This data then could be used for PR. Therefore, experience from the German inpatient sector (see above) could be used.