Publicly accessible online review data were obtained from HealthGrades, a well-known patient review website in the United States. Among many other sites, we focused on HealthGrades for two reasons. First, HealthGrades is widely used by patients who receive healthcare services from a full range of medical specialties in the United States. Second, HealthGrades provides a well-organized sitemap structure that facilitates data extraction. We analyzed data from a single review site because the data structure and measures of dentist demographics and performance vary, which will hinder data consolidation. In addition, data from multiple patient review sites would have little impact on representation and generalizability of this study because an active dentist typically has profiles on all popular patient review sites.

Online reviews for 204,751 dentists were extracted. This census approximates but does not fully match the workforce statistics (199,486 working dentists as of 2018) reported by the American Dental Association (ADA) [22] because some profiles are for recently inactive dentists and misclaimed or inappropriately captured profiles. The data contain the following attributes: state, city, specialty, gender, age, language, education, number of reviews, ratings, reviews, and wait time.

There were 41 dental service listings in HealthGrades. We categorized these services into the 10 dental specialties defined by the ADA, general dentistry, and others (ie, unidentified and miscellaneous), resulting in 8 specialties (ie, dental anesthesiology, endodontics, oral and maxillofacial pathology, oral and maxillofacial surgery, orthodontics and dentofacial orthopedics, pediatric dentistry, periodontics, and prosthodontics), general dentistry, and others for downstream analyses. Public health dentistry and oral and maxillofacial radiology were excluded as they have only one entry at most. The data extraction was completed in September 2019. The study was identified as a nonhuman study by the Institutional Review Board of the University of South Carolina.

We employed statistical analyses to assess the associations between ratings and dentists’ characteristics using R Project for Statistical Computing. We used descriptive statistics to calculate proportions and mean distributions. Based on reported online reviews studies in general medical specialties, we hypothesized that (1) female dentists, (2) young dentists, and (3) short wait times would be associated with higher overall ratings. We also hypothesized that specialties are associated with overall ratings. An independent sample t-test was used to test whether ratings differ by gender. Analysis of Variance (ANOVA) was used to test whether ratings differ by specialties, age, and wait time, respectively. We used Hedges g to approximate the effect size because of unbalanced sample sizes in comparison groups.

Semi-automated natural language processing was used to identify concepts related to patient experience with limited human labor required. Figure 1 is a diagram of text mining procedures. We used Python for text processing and computation. Built on our pilot study of analyzing patient-generated reviews [23], we first extracted reviews from online sites, followed by standard data cleaning procedures, including tokenization and removal of stop words.

We then calculated bigram and trigram collocations. Collocations are habitual expressions of multiple words. In this study, we ranked using the “likelihood ratio” method [24], top 200 bigram and trigram collocations, respectively, from reviews associated with every rating category (ie, 1, 2, 3, 4, 5).

We observed a number of the collocations that are irrelevant to concepts of dental care and patient experience (eg, “phone call” and “many years ago”) but were still ranked top 200 by likelihood ratio. Therefore, two raters (YL and CL) independently picked collocations related to patient experience using a 4-point Likert scale (“definitely relevant,” “somewhat relevant,” “somewhat irrelevant,” “definitely irrelevant”). Inter-rater reliability was assessed using Cohen kappa. Two raters discussed on the collocations that received contrary opinions (relevant vs irrelevant) until a consensus was reached.

Next, we mapped patient-experience-related concepts onto a total of 17 composite measures from the Patient Experience Measures from the CAHPS Dental Plan Survey. Table 1 shows the 17 measures categorized in three dimensions. Two reviewers (RS and JT), who are professors of dental medicine and licensed dentists, independently completed the mapping procedures. A consensus was reached after a discussion of the initial mapping results.

Table 2 shows the demographics of these dentists. Among the collected data, 154.683 (75.55%) dentists received at least one rating or review. The number of reviews ranged from 1 to 1789. There were more male dentists than females with a percentage difference of 85.87%. The majority of dentists (78.55%) were general dentists, followed by dentists specialized in orthodontics and dentofacial orthopedics (9.23%), pediatrics (4.76%), and endodontics (3.81%). The majority of dentists (91.83%) did not specify any second languages other than English. Among those who indicated speaking a second language, Spanish, Hindi, Arabic, French, and Chinese were most common. There were 150,571 (73.54%) dentists who received an overall rating of ≥3 out of 5, and 50,068 (24.45%) of all dentists received no ratings.

ANOVA showed no significant effect of specialties on ratings (F_{9, 154673}=58.74, P=1.37, g=0.06). However, the average rating was higher for female dentists (M=4.58) than male dentists (M=4.57) (t₇₁₈₈₁=2.45, P<.01, g=0.01). See Figure 2.

We also found a significant effect of age on ratings (F_{7, 107128}=246.97, P<.001, g=0.11) with younger age associated with higher ratings. The Tukey Honestly Significant Difference (Tukey HDS) test showed a significant difference in ratings between each pair of age groups except “30-39” vs “Over 89” (P=.15, g=0.01), “Under 30” vs “30-39” (P=.10, g=0.01), “40-49” vs “Over 89” (P=.95, g=0), “50-59” vs “60-69” (P=1.00, g=0), “50-59” vs “Over 89” (P=1.00, g=0), “60-69” vs “Over 89” (P=1.00, g=0), and “80-89” vs “Over 89” (P=1.00, g=0). See Figure 3.

There was also a significant effect of wait time on ratings (F_{4, 150055}=10417.77, P<.001, g=0.26) with shorter wait times associated with higher ratings. Tukey HDS showed statistical significance in comparing the mean difference of ratings in each pair of wait times (all P<.001, g=[0.01, 0.18]). See Figure 4.

Cohen kappa (equally weighted) was 0.95 between the two raters who independently identified patient experience-related words and phrases from 2000 automatically extracted collocations. After discussion, they identified 29 words and phrases, then two other reviewers of dental experts independently mapped the 29 words and phrases onto the 17 composite measures in the Patient Experience Measures from the CAHPS Dental Plan Survey. Figure 5 shows a map of links. Each composite measure has 2-10 representing words and phrases. Out of the three dimensions of the patient experience, there were more topics representing “care from dentists and staff” and “access to dental care” compared to “dental plan costs and services.” Patients were more likely to discuss their experience with dentists and staff than health insurance providers. There were eight words and phrases related to the patient experience that did not correspond to any composite measures. Some of these topics were specific to dental care. For example, discomfort (eg, painful/painless root canal or deep cleaning) is a common type of feedback from dental patients. Topics relating to ethics (eg, high-pressure sales and unnecessary dental work) have received little attention in dentistry but merits further research.

Over the last few years, researchers have begun a systematic analysis of online patient reviews. In the United States, several empirical studies investigating online reviews of general healthcare services and specialties are well documented, but such studies have not been performed in dentistry [1]. To the best of our knowledge, this study represents the first study of online reviews of dental care in the United States. In particular, this study demonstrated that online patient reviews are an essential data source for studying dental patient experience. This study also characterized patient feedback to dental care, which can inform dental care quality improvement.

We found several factors associated with overall dentist ratings. In particular, female dentists were rated slightly higher than their male counterparts, although this should be interpreted with the understanding that the effect size is small (g=0.01), and the sample size was small for senior dentists. The gender and age differences we found in this study were similar to a study of online reviews of dentists in Germany [25] as well as reviews of surgeons [26,27], but such a statistical significance was not consistent in other studies. Earlier studies on patient-provider communication suggested that female healthcare providers engage in significantly more active partnership behaviors, positive talk, and building trust with patients [28]. Female providers were also found associated with lower mortality and readmission as compared to their male counterparts [29]. The literature on healthcare quality suggests that younger physicians in acute care hospitals report lower mortality rates [30]. Our data also showed that higher overall ratings were associated with shorter wait times (g=0.26). This finding has been reported and discussed in several studies of online patient reviews [1]. Patient review websites such as HealthGrades and RateMDs have included “wait time/punctuality” as a default measure for healthcare providers.

The proliferation of patient review websites represents a wealth of patient-experience data, but these data remain understudied. In this study, we identified unstructured descriptions of patient experience using a method integrating quantitative text mining and qualitative human evaluation. Our method recognizes the role of automated textual data analytics in harnessing information from online reviews, consistent with other recent studies [31-37], although some researchers argued the limitations of text mining because it involves limited supervision by human experts [31]. To minimize the impact of this lack of human oversight, we incorporated human evaluation procedures into the automated natural language processing effort. Our findings showed that patient reviews covered a full range of topics measured by the Patient Experience Measures from the CAHPS Dental Plan Survey and demonstrated a high level of correlation. Among the eight topics not corresponding to any measures from the survey, some topics may have provided a nuanced view of dental care as compared to patient survey responses [36]. These findings suggest that online patient reviews can be used to assess patient experience during dental care.

Our study has the following limitations. First, data from Healthgrades, like data from any other patient review website, may be incomplete and biased. Not all dentist profiles have been claimed. Although authentication is required for dentist profile information, inaccuracies may still exist. The overall ratings may be biased because patients who are happy with health services are more likely to leave ratings and reviews [2]. Second, although there are good reasons for using data from a single review site, this limitation may still weaken the reliability of the study. Repeated measures and follow up studies are needed to evaluate the findings of the this study. Third, it is challenging to differentiate the descriptions of the patient experience from patient satisfaction as patients often write about their satisfactory or unsatisfactory experience with emotions and personal preferences. In this study, we only analyzed words and phrases at a semantic level, while contextual information is limited. Third, text mining is efficient for big data analyses but falls short in domain-specific and context-based analyses, in which traditional qualitative approaches should be considered as an essential complement of text mining [31].

Despite these limitations, this study analyzed an extensive dataset and found an association between dentists and online patient reviews. The thematic analysis also identified themes of patient experience similar to those of CAHPS, suggesting that online patient reviews can inform improved quality in dental care.

This study demonstrated that PORs are a potential data source that can supply rich performance data from the patient perspective, based on which assessments of dental care quality and the patient experience is feasible. We also identified several factors associated with dentists’ overall ratings, which could be used to inform dental care quality improvement.