We used the Yelp application programming interface (API) between October 1 and October 20, 2018, to obtain a list of facilities that were tagged as “urgent care” in the United States. We also retrieved the star ratings for all reviews that were associated with these facilities. We removed all facilities that did not have a US zip code. We were unable to obtain the full texts of these reviews—Yelp's terms of service prohibit web scraping, and Yelp declined to provide us with permission to use the text. Thus, by using the list of facilities that was obtained from the Yelp API, we ran the data collection algorithm from January 21 and February 10, 2019, and obtained all of the Google Maps reviews of each urgent care facility that were posted before January 21, 2019. Unlike Yelp, Google's terms of service permit the collection of their reviews as long as it does not put undue burdens on Google's servers. To collect data from Google Maps reviews, we designed a computer program (Multimedia Appendix 1) for automatically collecting these reviews based on their Google Maps URLs. The Google Maps URLs of the urgent care facilities were collected after we presented their titles and addresses (obtained from the Yelp API) to 20 workers on Amazon Mechanical Turk, a crowdsourcing marketplace. These 20 workers were selected based on their prior demonstrated ability to successfully collect a set of 100 known URLs. Workers received US $0.09 for each URL collected. To summarize the contents of web-based reviews, we used a machine learning algorithm that was designed to summarize text—the latent Dirichlet allocation (LDA) topic model [22]. This algorithm was implemented in the LDA python package [23] (default settings were used) to the fit the model to the Google Maps data set. Topic models identify review topics automatically without human intervention by examining the word co-occurrence statistics within each review [24]. For each topic, we calculated the total number of word tokens that were found in positive reviews (4 or 5 stars) and negative reviews (1 or 2 stars). We fitted 3 topic models (one model with 10 topics, another with 20 topics, and another with 50 topics) to the data. We selected the model that generated the most coherent topics without a large increase in perplexity (a measure of model goodness of fit that is commonly used in natural language processing; Multimedia Appendix 2). Afterward, we extracted antibiotic-related reviews by using a list of keywords that was generated by one of the authors (RH)—a pediatrician who specializes in antibiotic stewardship (see Multimedia Appendix 3 for a keyword list). For each review, we examined the proportion of words in each topic. We applied the same procedure to the subset of reviews containing antibiotic-related keywords. We then developed a qualitative codebook to determine the content and sentiment of 200 reviews that were sampled at random from all reviews containing these keywords. This study was approved by The George Washington University Committee on Human Research Institutional Review Board (Federal Wide Assurance number: FWA00005945; institutional review board registration number: 180804).

Four authors (LR, MF, MC, and SD) affiliated with The George Washington University Antibiotic Resistance Action Center collectively reviewed a subset of the 200 randomly sampled reviews to determine the types of messages that were present in the reviews [25]. After this initial review and the development of inductive codes, the reviews were categorized into one of the following categories in the codebook: (1) the Yelp effect category (the patient wanted antibiotics but did not receive them); (2) the opposite of the Yelp effect category (the patient received antibiotics but did not want them); (3) the convenience, inconvenience, and wait times category; (4) the staff competence or incompetence, courtesy and attitude, and satisfaction of care category; (5) the cost and price of drugs per visit (including sticker shock) category; (6) the other prescription-related complaints category; and (7) the other or none of the above category. Additionally, all reviews were annotated as “positive” (eg, the patient was satisfied with their care, and the review had 4 or 5 stars) or “negative” (eg, the patient was dissatisfied with care, and the review had fewer than 4 stars).

The four annotators then independently reviewed the same 200 randomly sampled reviews to assign them to 1 of the 7 categories. The final categories for the reviews were assigned based on the majority category among annotators. If there was no majority category, disagreements were resolved discursively until a consensus category was agreed upon.

Some reviews mentioned that the reviewer did not receive an antibiotic when it was expected, even if that was not the main message. Thus, after assigning each message a primary category, the same four annotators revisited all 200 reviews to determine if they mentioned the Yelp effect in passing (ie, whether the review mentioned an unfulfilled expectation for antibiotics). A code for mentioning the presence or absence of the Yelp effect—even if it was mentioned in passing—was then assigned to each review as a secondary code. By using these 200 annotated samples, we inferred population proportions for each category in our codebook and calculated 95% CIs.

By using the Yelp API, we identified 8662 unique urgent care facilities that had 84,127 unique reviews. We collected 481,825 US-based reviews from Google Maps. Of these, 340,328 (70.63%) contained some text. The average star rating in Yelp reviews was significantly lower than the average star rating in Google Map reviews (t_565,950=82.38; P<.001).

Figures 1–4 display the distributions of the number of reviews and mean reviews stars from both Yelp and Google Maps on a 2-by-2 map. The distributions show an apparent difference between various geographical regions of the United States. The biased distribution is intriguing and can prompt many other research questions. Therefore, all state-by-state maps featuring the same information are being hosted on The George Washington University cloud, which can be made available to other researchers upon request.

We categorized the 481,825 reviews with text from Google Maps into the following three groups: (1) reviews without any text (n=141,497; star rating: mean 4.53); (2) reviews with text that did not mention antibiotic-related keywords (n=332,566; star rating: mean 3.94); and (3) reviews with text that mentioned antibiotic-related keywords (n=7762; star rating: mean 2.40; Figure 5). We found significant differences in the average star ratings across these three groups; a posthoc Tukey honestly significant difference test for multiple comparisons showed that reviews with antibiotic-related keywords had the lowest ratings compared to those in reviews without text (mean difference=−2.13; P<.001) and reviews with text but no keywords (mean difference=−1.54; P<.001). Reviews without text were significantly more positive than reviews with text but no keywords (mean difference=0.59; P<.001; all descriptive statistics are in Multimedia Appendix 4).

The LDA topic modeling analysis (Figure 6; Multimedia Appendix 2) yielded 20 topics. The most negative topics pertained to rude staff, wait times, billing, callbacks, and other aspects of customer experience. Although the “infections and symptoms” topic was also predominantly negative, this topic did not make up a plurality of the reviews (481,760/6,795,468, 7.09%). With regard to topics with antibiotic-related keywords, Figure 7 shows that the most common topic pertained to infections and symptoms and was predominantly negative; however, several other topics pertaining to customer experience were also predominantly negative.

The annotators who labeled the seven inductive content categories achieved moderate reliability (Fleiss κ=0.42) with the first set of 100 reviews. All annotators agreed on sentiment. After disagreements were resolved, a second round of annotation for the next set of 100 reviews yielded substantial agreement (Fleiss κ=0.65), and disagreements were again resolved by consensus among reviewers. Table 1 summarizes the results of these annotations. Of a total of 200 reviews, we found that only 5 reviews (2.5%; 95% CI 0.3%-4.7%) exhibited the Yelp effect as the primary message category. By applying CIs to the full set of 8078 reviews containing antibiotic-related keywords, we expected that between 27 and 377 reviews would exhibit the Yelp effect as the primary message with 95% confidence. Thus, in our data set of 481,825 reviews, at most, 377 (0.08%) were expected to exhibit the Yelp effect.

The annotators also reexamined all reviews to determine if they mentioned a Yelp effect in passing. We found that of a total of 200 reviews, 42 (21%) had some mention of reviewers not having received antibiotics when they were expected (Fleiss κ=0.75). Thus, with 95% confidence, between 1240 and 2152 of our 8078 reviews with antibiotic-related keywords exhibited the Yelp effect, even if it was only mentioned in passing. In our data set of 481,825 reviews, at most, 2152 (0.45%) were expected to exhibit the Yelp effect.

Our data suggest that the Yelp effect is quite rare. Out of a set of almost half a million reviews (N=481,824), fewer than 1 in 1250 (0.08%) seemed to contain the Yelp effect as the primary message (with 95% confidence). Furthermore, with 95% confidence, fewer than 1 in 225 (0.45%) reviews seemed to contain the Yelp effect as the primary or secondary message.

In contrast, we found that (138/200) 69% of the reviews in our annotated data set focused primarily on assessments of staff competence and the quality of personal interactions. This suggests that in terms of the extent that a Yelp effect exists, patients express this effect by questioning the expertise or personal qualities of urgent care staff. This may put urgent care providers in a bind; although they should not prescribe antibiotics inappropriately, a failure to explain to patients why the patients' preferred treatment is ineffective may lead to reviews that are designed to undermine care providers' credibility, expertise, and personal qualities. Thus, it is of paramount importance that both care providers and urgent care staff provide high-quality care and leave patients with a meaningful understanding of why they received the treatment that they did. For example, prior work has shown that patients' expectations for antibiotics are associated with categorical gist perceptions of the risks and benefits of antibiotics [26-28] and that patients more likely to be satisfied when they understand the gist of appropriate prescribing. This promotes the need to better communicate rationales for prescribing antibiotics in a manner that enhances patients' insights into why decisions are made and, by extension, their assessments of care providers' competence. Naturally, care providers' attitudes toward patient care are also important.

We aimed to answer the following question: is there a Yelp effect? The 2.28% (7762/340,328) of Google Maps reviews that mentioned antibiotics were indeed significantly more negative than those without antibiotic-related keywords (P<.001). Furthermore, our results show that reviews of urgent care centers on Yelp are significantly more negative compared to those on Google Maps (P<.001). Thus, we cannot rule out the existence of a Yelp effect on either Yelp or Google Maps. However, our results show that antibiotic prescription is merely one of the many potentially addressable issues in doctor-patient communication and may not be the primary source of negative web-based reviews. Indeed, patient satisfaction seems to have been most strongly linked to customer service issues (eg, wait times, rude staff, billing practices, etc). Thus, we must question whether claims regarding the impact of antibiotic prescriptions on negative reviews of urgent care centers are exaggerated. In recent years, some authors have suggested the presence of an effect that is similar to the Yelp effect in the context of opioid prescription [29,30]. However, similar to our findings, other studies have shown that these negative reviews are primarily comments on physicians’ attributes or administrative attributes [31].

The limitations of our work include our inability to hand-annotate all of the 481,825 reviews in our data set. Instead, we annotated 200 of the 7762 (2.58%) messages that were identified to have antibiotic-related keywords. This limitation was mitigated by the fact that these 200 messages were selected uniformly at random, meaning that they are likely to be representative of messages with antibiotic-related keywords. It is possible that our choice of keywords might have introduced selection bias; specifically, we assumed that patients who expected to (but did not) receive antibiotics would have said so in their reviews. Thus, we cannot rule out the possibility that patients were insincere when providing their reasons for negative reviews. However, our findings clearly indicate that patients were willing to express dissatisfaction with several other topics that do not directly pertain to antibiotics. Web-based reviews also often lack key patient information (eg, visit reason, medical history, and demographics). Finally, we do not claim that our results generalize beyond urgent care settings.

Our analysis shows that the Yelp effect may not be a major driver of negative sentiments in web-based reviews. Rather than compromise medical and public health recommendations by acceding to the potentially faulty perceptions resulting from patients' desires, urgent care facilities should instead invest in efforts for improving patients' overall experience, such as reducing wait times, making billing practices transparent, and investing in training staff members to adhere to the best standards of customer service. Although these steps may not prevent all negative reviews, our analysis suggests that antibiotic prescribing need not be the focal point for patient satisfaction in urgent care settings.