On the basis of the research objectives mentioned above, we listed the following search criteria before we started the literature search. The inclusion criteria were as follows: (1) studies that focused on PORs of physicians or hospitals, (2) studies that reported qualitative or quantitative results from analyses of PORs, and (3) peer-reviewed studies written in English. The exclusion criteria were (1) studies that did not report empirical outcomes from analyses of PORs and (2) editorials, reviews, or commentaries. Excluded studies were, for example, focused on physicians’ responses to online reviews [19], reported innovative methods for analyzing PORs without reporting the analytical results [20,21], or focused on characteristics of the patients who had used PRWs without reporting POR-related outcomes [2,22].

Following the principles of Preferred Reporting Items for Systematic Reviews and Meta-Analyses [23], we searched the major databases of PubMed, EMBASE, CINAHL, and Science Direct in January 2019. Search terms from previously published studies were used [17,18,24], including rating sites (websites), review sites (websites), online reviews (ratings), doctor (physician and hospital) ratings, and patient reviews (ratings). As shown in Figure 1, the initial search identified 2837 articles. After reviewing the titles and abstracts to determine relevance and removing duplicates, 90 articles were further reviewed by reading full texts. A total of 48 articles that met the inclusion and exclusion criteria were identified for detailed review. Next, we searched the reference sections of the 48 articles and consulted experts in the field to identify additional articles by hand search, resulting in 26 additional articles for review. After removing duplicates, we identified 69 articles or 63 studies to include in the review. Articles that reported findings based on the same data source, similar design, and research questions were counted as one study. The systematic review protocol was registered in PROSPERO: International Prospective Register of Systematic Reviews.

A total of 2 researchers independently reviewed all articles and extracted the following information using a predesigned table: authors and publication year, study time and location, PRWs used in the study, type of providers being studied, number of PORs and providers being analyzed, study design and analytical approach, and key findings. The intercoder reliability, calculated by using Cohen kappa, was 0.86. In the key finding analysis, the researchers listed the bullet points of major findings from each study and discussed the discrepancies until a consensus was reached. Owing to the heterogeneity of the studies, no study appraisal was carried out. This review was not focused on a single health outcome; instead, we aimed to identify and synthesize available POR studies, and no meta-analysis was conducted.

A total of 63 studies (69 articles) were included in the reviews (Multimedia Appendix 1) [1,3,10,11,25-85]. Although PRWs have been available for more than 2 decades, the earliest study on PORs was published in 2009 [25], and most of the studies (61/63, 96.8%) were published after 2010. Out of 63 studies, 48 were conducted in the United States, including Puerto Rico, 5 in Germany [26-31], 3 in the United Kingdom [32-34], 3 in China [35-38], 3 from the Netherlands [39-41], 1 from Australia [42], and 1 from Canada [86] (Table 1).

Most studies (36/63, 57.1%) retrieved PORs from multiple PRWs with 27 studies (27/63, 42.9%) using a single PRW for data analysis. Some earlier studies googled “patient review” or providers’ names to retrieve PORs, and the most popular or promoted PRWs emerged through such an online search. The PRWs used in these research studies varied across countries. For example, in the United Kingdom, most physicians and hospitals were rated on the National Health System Choices website, which was a single PRW used in the studies from the United Kingdom [32-34]. The most popular PRW in Germany was Jameda [28-31], whereas HaoDF was used in China [35-38]. In the United States, a large number of PORs have accrued on the most popular PRWs including generic consumer review sites such as Yelp, Yahoo, and Google, as well as specialized PRWs such as RateMDs, HealthGrades, and Vitals (see Figure 2).

Studies that compared multiple PRWs found a low correlation between these sites [43,44]. For example, Nwachukwu et al reported that the correlations (r) between PRWs were 0.32 approximately 0.51, P<.001[43]. Physicians on one PRW were rated differently on other PRWs, whereas no PRW contained all consensus core domains of quality measures [45]. Some studies questioned the reliability of PRWs given that most physicians only have a very small number of ratings [10,46].

Out of the 63 studies, 17 (17/63, 27%) reported PORs of HCOs, including hospitals, urgent care centers (emergency departments), and nursing homes [27,32,34,39,40,44,47-53,87,88], and 55 (55/63, 87.3%) were focused on clinicians. Of the 55 studies that reported PORs of clinicians, 14 (14/55, 25.5%) included multi-type physicians (general practitioners and specialists), 2 (2/55, 3.6%) were focused on general practitioners [33,54,55], 1 (1/55, 1.8%) reported physical therapists [88], 2 (2/55, 3.6%) reported dentists [30,88], and the remaining 36 (36/55, 70.6%) were focused on specialists, including surgeons, dermatologists, urologists, Ob/Gyns. Of these 36 studies on specialists, 21 (21/36, 58.3%) were focused on various types of surgeons (see Figure 3).

The number of health care providers reviewed ranged from 20 to 212,933 with a median of 600. The number of PORs analyzed ranged from 30 to 2,685,066 with a median of 5439. The number of PORs included in the analyses has grown substantially in the past 9 years.

Not all physicians had an online rating. In Germany, only 37% of physicians were rated online [29]; a recent study in 3 metros of the United States reported that 34% of physicians did not have any PORs and most physicians still had no more than 1 review [10]. Even among those physicians with online reviews, the number of PORs varied significantly across specialties. For example, 96% of cardiologists in the United States were rated online [56]; 25% of the hospitals included in Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) had Yelp ratings [51]. Specialists were twice as likely to be rated online than general practitioners [26]; radiologists and pathologists were least likely to be rated online [42].

A considerable number of studies (27/63, 42.9%) were descriptive and reported only frequency analyses, including the average numbers of ratings per provider, the percentages of providers that have been reviewed online, and the mean scores of PORs. Studies that focused on HCOs and specialists typically identified the providers from a directory before searching for their PORs. In contrast, studies that focused on all types of providers typically retrieved PORs directly from PRWs without a preselected list of providers.

A total of 19 (30.2%) studies analyzed the narrative comments of PORs. Previous studies in this regard used traditional qualitative methods to retrieve major themes from these comments [3,25,32,54,55]. Recent studies have applied more advanced techniques such as natural language processing (NLP). For example, topic models, such as Latent Dirichlet Allocation, have been used as an efficient tool to automatically cluster POR comments by topics [35,36,38,50,87]. The use of such advanced analytical methods enabled content analysis of hundreds of thousands of narrative comments.

More than half of the studies (n=38/63, 60.3%) employed a comparative design. They typically compared PORs with (1) traditional surveys of patient experience [27,34], (2) providers’ characteristics [28,29,33,34,57,58], (3) clinic outcomes such as patients’ readmission rates or mortality rates [52,56,59], and (4) traditional “golden standards” of health care quality indicators (eg, HCAHPS structural and quality of care measures) [39,44,45,47,50,53]. Furthermore, 1 study compared PORs between China and the United States [38] and 2 studies compared PORs across different PRWs [43,44].

Most patients made positive comments about their providers and would recommend their providers to families and friends. Out of 63 studies, 27 studies (27/63, 42.9%) reported mean scores of PORs ranging from 2.37 to 4.51 (out of 5) with a median value of 4 and a mean value of 3.89 (unweighted).

The studies that analyzed the patients’ narrative comments found that these comments covered the entire health care encounter of the facility and staff [54,55], including physicians’ demeanor, staff friendliness, empathy, and cost [60,61]; patients also cared about the ease of scheduling, time spent with patients, and wait time [62].

The existing studies that compared PORs with characteristics of providers found that physicians with higher ratings had the following characteristics: (1) female, young age [29,43,46]; (2) more online presence [58]; (3) board-certified with extensive training experiences and graduated from a highly rated medical school [1,89]; (4) active status and years in practice [63]; (5) specialties [37,88]; and (6) locations [86]. However, some studies found no interactions between PORs and either genders, regions, or academic proclivity [46,64,65]. Furthermore, 1 study found surgeons with higher volume of procedures had higher POR ratings and better comments [59]. Patient characteristics also affected PORs. For example, female, seniors, and patients covered by private insurance were more likely to provide positive PORs [27,30].

As summarized in Table 2, convergent findings suggested a strong association between PORs and traditional patient satisfaction surveys. For example, several studies found moderate-to-high degrees of correlation between PORs and HCAHPS patient experience measures [31,33,34,53,66] and the Press Ganey Medical Practice Survey for patient satisfaction [67], respectively. Content analysis studies also reported a considerable overlap between the narrative comments of PORs and thematic domains of HCAHPS surveys [47,50,51]. Some of these studies also identified additional domains not included in HCAHPS surveys. [50,51]. Similar findings of correlation tests were reported from studies in Germany [31] and the United Kingdom [33,34]. Furthermore, 2 studies from the Netherlands reported that hospitals under supervision or inspection from authorities had lower POR ratings [39,40].

Table 2 also includes the summaries of the relationship between PORs and clinical outcomes and other quality measures. Most of the 9 studies on the relationships of PORs and clinical outcomes reported weak or no relationship [11,31,33,52,53,56,66,68,69]. For instance, a study of PORs on cardiologists found no correlation (Spearman ρ=−0.06; P=.13) between PORs and mortality rates following the coronary artery bypass surgery [56]. Similarly, Greaves et al found a weak correlation between PORs and clinical outcomes of providers in the United Kingdom (Spearman ρ=−0.18 approximately 0.18; P<.001) [33]. By contrast, Bardach et al compared PORs of hospitals in Yelp with quality measures from HCAHPS and found that higher scores of PORs were associated with better clinical outcomes, including lower mortality and readmission rates [53]. Studies also reported significant but low degrees of association between PORs and (1) patient likelihood of visiting their primary care physicians within 14 days of discharge [11], (2) cost of care [11,66], (3) 30-day readmission and length of stay [69], and (5) other hospital level CMS quality measures [44].

Our summaries of the existing 63 studies on PORs revealed that the number of health care providers (including clinicians and HCOs) being reviewed represented only a small number of the total health care workforce. The number of reviews per clinician varied from zero to hundreds, indicating a very skewed distribution in these PORs. As compared with general practitioners, specialists, especially surgeons, were more likely to be reviewed and included in the analyses of PORs. Overall, the online ratings and comments were positive. Only a small number of studies compared the correlations between PORs and patient satisfaction and clinical outcomes. These studies suggested that PORs were highly correlated to the “patient experience” measured by traditional patient surveys. Nevertheless, there were inconclusive findings on whether PORs were inconsistent with traditional measures of clinical outcomes. Notably, reviewed studies have identified several domains of patient experience that were not covered by the traditional patient surveys, for example, HCAHPS [50,51].

The current literature on PORs suggests a relatively new but fast-growing field. The number of published studies was small when compared with the exponential growth of PORs. Therefore, we have made the following recommendations for future studies on PORs.

First, studies with rigorous design, longitudinal nature, and larger samples are needed. POR studies present challenges of data acquisition and processing because of the nature of large and heterogeneous online data. The latest Web crawling techniques have enabled efficient retrieval of large quantities of POR data. Advanced analytical techniques such as machine learning and NLP can be employed to expedite large-scale analysis of PORs.

Second, most existing studies are focused on specialists in metropolitan areas [10,70-72]; more studies are needed to understand other disciplines of health care providers and those who serve in nonmetro areas. Studies that identify consumer-based assessments for underrepresented types of HCOs, such as nursing homes, public health services, and substance treatment centers, are minimal or missing in the literature. There was only 1 study that reported PORs for nursing homes [49]. Many of these HCOs serve vulnerable populations who are not typical PRW users, but their family caregivers and other advocates may also provide valid PORs.

Third, we anticipate more studies that go beyond the simple descriptive analysis and test theory-based hypotheses to provide more clinical and policy implications. In recent years, we have observed emerging studies that compared PORs with traditional measures of patient experience and clinical outcomes. However, the current literature is limited in terms of a lack of consistent POR reporting and insufficient advanced statistical analyses of POR data and their relationship with quality measures. We call for more empirical studies with meaningful hypotheses, rigorous design, and appropriate data analytics.

Finally, we have observed that PORs have begun clustering on a small number of popular PRWs (Figure 2). With the recent announcement of Amazon’s entrance into health care [90], online reviews by health care consumers may become even more clustered. Whether and how the clustering of PORs on the growing dominance of commercial PRWs would affect consumer health behaviors and health care quality remains unstudied.

The growing body of literature on PORs indicates its increasing importance in patients’ decision making, which provides policy and practice implications for health care providers, patients, PRW owners, and policy makers.

Notably, health care providers should not underestimate the importance of PORs. Instead, they should recognize the importance of PRWs for their “digital brand” and stay aware of the PORs posted to popular PRWs [91]. Physicians can use anonymous PORs for the evaluation of patient satisfaction and assessment of patients’ need. In addition, friendly and personalized responses to PORs may enhance positive patient-provider communication [19].

From a consumer’s perspective, patients need to know that only a small number of physicians have been reviewed online and the average rating score for a physician might not be sufficient for choosing a doctor as assumed, given the tendency of consumers to provide feedback on experiences that are unusually positive or negative. As posting the health care experience becomes more commonplace, we anticipate a “consumer’s guide” to help patients navigate the PORs and make more informed choices [92,93].

For PRW owners, as PORs are often unstructured, not adjusted for risks, and unverifiable, they should take more social responsibilities by adding design components to enable identity authentication, to remove inflammatory or abusive comments, and to assist patients on how to use PRWs to avoid misinformation [3,4,94]. We also call for a consistent rating scheme to facilitate the evaluation of providers using data from various PRWs.

For policy makers, the question of whether PORs can be used as an indicator of health care quality is still controversial; policy makers and health care providers should acknowledge and embrace its increasing importance for patients [7,95]. The PORs can reflect instant feedback of patients’ medical encounters, the context of their ratings, and what they truly value. Some of the constructs of patient experience identified from analyzing PORs can be used to strengthen or complement the current measures of health care quality and to provide rapid recognition of quality perception gaps along with service corrections or other proactive quality interventions when needed [96]. Although we recognize the growing weight of PORs in consumer health behaviors and the potential of applying PORs in improving health care quality, we call for broader collaborations of key stakeholders, including patients, caregivers, health care providers, PRW owners, policy makers, and health services researchers, to engage in conversations and joint efforts to construct a positive patient-provider feedback loop.

Some potential biases need to be noted while interpreting the results from this review. First, this review was focused on the published studies that analyzed PORs, so the findings related to PORs only reflected those published studies but not the whole picture of PORs. Given the vast and ever-growing number of PORs, only a small fraction was studied and published. Second, only a small number of patients would actually provide ratings of their medical encounters. These motivated patients are more likely to be younger, female, living in metropolitans, and spending more time online [4]; thus, there is a potential bias in the existing PORs. These biases are not methodological flaws in conducting the systematic review but require caution when interpreting study findings.

In addition to the above potential biases, we should also note the limitations of the study. Though we tried our best to thoroughly search the major databases, it is possible that some relevant studies were missed. As we concluded the search in January 2019, a few recently accepted papers were not included. Our search was limited to peer-reviewed literature; we may have missed some gray literature that is equally important for the POR research. Additionally, because our review was limited to the literature published in English, the review did not cover articles published in other languages. Finally, because of the heterogeneity in outcome reporting and study design, we did not carry out an appraisal of study quality. The number of PORs ranged from a few dozens to hundreds of thousands and the ratings were based on different scales, so we did not conduct a meta-analysis.

To conclude, the current body of the peer-reviewed literature on PORs is still small but growing rapidly. We found that overall PORs tended to be positive, and the narratives of PORs have provided insights into multiple domains of patient experience and health care quality. We call for more research on PORs using rigorous design and large samples along with better use of POR information by patients, physicians, and policy makers. We also advocate for recommendations or guidelines of POR use to help patients make informed choices and foster the application of PORs for improving health care quality.