This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
There is a push towards quality measures in health care. As a consequence, the National Committee for Quality Assurance (NCQA) has been publishing insurance plan quality measures.
The objective of this study was to examine the relationship between insurance plan quality measures and the participating providers (doctors).
We collected and analyzed provider and insurance plan data from several online sources, including provider directories, provider referrals and awards, patient reviewing sites, and hospital rankings. The relationships between the provider attributes and the insurance plan quality measures were examined.
Our analysis yielded several findings: (1) there is a moderate Pearson correlation (
Our findings may be used by consumers to make informed choices about their insurance plans or by insurances to understand the relationship between patients’ satisfaction and their network of providers.
There are several health insurance marketplaces and search portals (eg, ehealthinsurance.com) that help individuals and small employers shop for, select, and enroll in high-quality, affordable health plans. Insurance plans are generally ranked based on relative quality and price. These marketplaces and search portals need to establish criteria and selection processes for quality measures. Most of them measure the quality of health plans by surveying plan enrollees on their satisfaction with their coverage and then publishing quality and satisfaction data online [
We collected a rich set of data for each provider ranging from average patient review scores, referral patterns, affiliated hospital scores, relative costs, and provider awards. Specifically, we used data collected from Centers for Medicare & Medicaid Services (CMS) and provider profile websites on a set of 600,000 US health care providers. We also collected ranking data from other sources; specifically, U.S. News was used for specialty-specific hospital rankings. We converted each provider’s information to a set of intuitive qualitative attributes. For instance, affiliated hospitals were mapped to specialty-specific rankings to assign a score to the affiliated hospitals of a provider relevant to their specialty. As a peer-nominated award, we selected the Castle Connolly award. Each year, Castle Connolly distinguishes top providers both nationally and regionally through a peer nomination process that involves over 50,000 providers, and hospitals and health care executives [
We then adopted a data-driven approach to determine if the provider attributes were correlated with the insurance quality indicators. Specifically, we measured the correlation between several provider attributes (reviews rating, awards, affiliated hospitals, etc) of member providers of an insurance plan to key quality scores of the insurance plans.
Key challenges to our data collection and analysis included mapping providers from CMS to providers in provider profile sites, mapping insurance names between accepted insurances obtained from provider profile sites and insurances obtained from NCQA, and mapping hospital names between each source. These challenges are due to the lack of a common identifier for providers, insurance plans, or hospitals across the data sources.
There have been several studies to determine the quality of health insurance plans. These studies can be split into two categories: (1) health insurance marketplaces and search sites, and (2) attributes associated with health plan quality.
There are several health insurance marketplaces, authorized by the Affordable Care Act, that help individuals and small employers shop for, select, and enroll in high-quality, affordable private health plans. In fact, the Affordable Care Act requires the US Department of Health & Human Services to develop quality data collection and reporting tools such as a quality rating system, a quality improvement strategy, and an enrollee satisfaction survey system [
Several surveys have examined the quality of health insurance plans based on consumer feedback and have tried to determine attributes associated with insurance quality. Feldman states that a cornerstone of high-quality integrated care for people with medical, behavioral, and long-term services and support needs is a dynamic person- or family-centered plan of care built on significant individual and caregiver involvement and comprehensive assessments and reassessments over time to capture changes in people’s circumstances and preferences. Other key ingredients identified were (1) a multidisciplinary care team with one accountable care coordinator, and (2) a comprehensive provider network with a strong primary care base and a range of other providers and services that can accommodate diverse needs throughout a lifetime [
URAC (Utilization Review Accreditation Commission), which is an independent, nonprofit organization known for promoting health care quality through its accreditation, education, and measurement programs, addresses the following key areas aimed at helping plans deliver safe, high-quality, patient-centered, high-value care: Wellness and Health Promotion; Care Coordination; Medication Safety and Care Compliance; Rewarding Quality; Care Delivery through a Network; Mental Health Parity; Measures—patient centeredness, coordination of care, patient safety, health plan administration, efficiency, effectiveness of care and health information technology integration; and Patient Experience of Care (Consumer Assessment of Healthcare Providers and Systems Survey) [
For the purpose of our data-driven analysis, we have collected a large amount of information about US health providers, mainly physicians, from multiple online sources including the CMS data on providers and hospitals, U.S. News rankings of hospitals, and additional provider information and reviews from provider profile websites. We have also collected information about the rankings of private, Medicare, and Medicaid health insurance plans from NCQA. We then mapped entities across sources to create a database of providers and health plans.
Visual description of data preprocessing.
Insurance information and patient ratings of providers were collected from both Vitals and Healthgrades [
Aggregating the datasets posed unique challenges for entity mapping, such as mapping providers from Healthgrades to providers in CMS, as described in the next section. In total, we collected information on 3.2 million distinct providers from CMS, 4600 distinct hospitals from CMS, 1.9 million distinct providers from Healthgrades, one million distinct providers from Vitals, and 1956 hospitals from U.S. News. We also collected information of 1264 health plans from NCQA. Of these, NCQA has ranked 1051 plans based on clinical performance, member satisfaction, and results from NCQA Accreditation surveys. The remaining insurances had partial data. After appropriate data transformations and entity mappings, we generated the set of provider attributes listed in
List of provider attributes used in our analysis based on the data collected.
Category | Attribute | Description | Source | Min. | Max. | Mean | Median |
General information | NPI | National Provider Identifier. | CMS | N/A | N/A | N/A | N/A |
Gender | Male or Female, as specified in the CMS data. | CMS | N/A | N/A | N/A | N/A | |
Specialties | A set of attributes, one for each specialty, eg, cardiologist. | CMS | N/A | N/A | N/A | N/A | |
From peers | NumReferrals | Normalized number of referrals. | CMS | 0 | 4018 | 70.1 | 10 |
Castle Connolly | Whether or not the provider is recognized by Castle Connolly as a distinguished provider. | Vitals | N/A | N/A | N/A | N/A | |
Average rating from patient reviews | UserRatings | Overall review score assigned by user (patient). | Reviews from Vitals and Healthgrades | 0 | 100 | 82.06 | 87.5 |
NumReviews | Number of patient reviews for the provider. | N/A | 0 | 247 | 0.96 | 0 | |
Insurance | NumInsurances | Number of insurers accepted by the provider. | Vitals and Healthgrades | 1 | 8 | 1.7 | 1 |
IndividualInsurers | A set of attributes, one for each insurer accepted by the provider, eg, Humana. | Vitals and Healthgrades | N/A | N/A | N/A | N/A | |
Hospital affiliations | HospitalRanking | The ranking of the provider’s affiliated hospitals. | CMS (hospitals) and U.S. News (ranks of hospitals) | N/A | N/A | N/A | N/A |
List of health insurance attributes used in our analysis based on the data allocated. All attributes in this table are from NCQA.
Category | Attribute | Description |
General information | PlanName | Insurance plan name. |
State | The state to which the plan belongs. | |
PlanCategory | The category of the plan, eg, private, Medicare, Medicaid. | |
PlanType | The type of the plan, eg, preferred provider organization (PPO), health maintenance organization (HMO). | |
Quality indicators – Overall | Rank | The overall rank of the plan. |
OverallScore | The overall score of the plan. | |
Quality indicators – Customer service | OverallConsumerSatisfactionScore | The score for consumer satisfaction. |
GettingCareScore | Scores based on appointments, preventive care, test, and easy and quick access to treatments. | |
SatisfactionWithPhysiciansScore | Scores based on providers, care revived and health promotion and education. | |
SatisfactionWithHealthPlanServicesScore | Scores based on handling claims and other plans services. | |
Quality indicators – Prevention | OverallPreventionScore | The score for preventive care. |
ChildrenAndAdolescentsScore | Scores based on well-child visits, immunizations, nutrition counseling, physical activity counseling. | |
Women’sReproductiveHealthScore | Scores based on prenatal checkup and postpartum care. | |
CancerScreeningScore | Scores based on various cancer screenings. | |
OtherPreventiveServicesScore | Scores based on flu vaccinations, chlamydia screening, and other preventive care. | |
Quality indicators – Treatment | OverallTreatmentScore | The score for different treatments. |
AsthmaTreatmentScore | Scores based on asthma medication and treatment. | |
DiabetesTreatmentScore | Scores based on blood pressure control, glucose testing and control, low-density lipoprotein cholesterol screening and control, monitoring kidney diseases. | |
HeartDiseaseTreatmentScore | Scores based on controlling blood pressure and cholesterol and beta-blockers after heart attack. | |
MentalAndBehavioralHealthScore | Scores based on depression medication, alcohol and drug dependence treatment, etc. | |
OtherTreatmentMeasuresScore | Scores based on monitoring key long-term medications, antibiotic use, testing for chronic obstructive pulmonary disease, etc. |
The names of insurance obtained from Vitals and Healthgrades differ from the names of insurance in the NCQA data. For example, “United Healthcare Services, CA” and “United Healthcare, CA” refer to the same insurance plan, as do “Aetna Life Insurance, AR” and “Aetna HMO, AR”. In order to achieve this mapping, we used the Levenshtein distance metric [
The hospital rankings listed by U.S. News categorize hospitals across several specialties for adults and children; for each hospital listed, the hospital’s score, name, and location were collected for each specialty for both adults and children. Further, the hospital specialties reported by U.S. News do not always correspond to the specialties listed by CMS. In particular, CMS uses a taxonomy of medical specialties that consider subspecialties, whereas U.S. News uses broad categories of specialties [
In order to identify Castle Connolly and patient reviews information for each provider, CMS providers needed to be mapped to Vitals and Healthgrades provider profiles. This mapping exercise allowed us to map 608,935 providers between CMS, Vitals, and Healthgrades, 25,514 of whom have received a Castle Connolly award. To map CMS providers to providers in the other sources (Heathgrades and Vitals), we followed a hybrid automatic-manual data integration approach. First, we identified a promising set of attributes to use for mapping, specifically, first name, middle name, last name, address, medical school, graduation year, affiliated hospitals, and specialties. For each attribute, we constructed a customized mapping algorithm. For example, the mapping between first names is computed using the Levenshtein distance between the two strings. Then, we assigned weights to each attribute matching score based on a large number of accuracy experiments, where the authors defined the ground truth mappings. We then computed a mapping threshold based on the mapping scores via more accuracy experiments. Note that each Vitals/Healthgrades provider is mapped to at most one CMS provider, so no duplicate provider data are present in the final dataset.
Only 4% of all mapped providers have received a Castle Connolly award, and 42% of all mapped providers have zero referrals. A majority of providers with zero referrals specialized in Internal Medicine, Family Medicine, or Emergency Medicine. Also, 213 of 1264 health plans collected had incomplete data. In order to correlate rank of affiliated hospitals and insurance scores, we needed the rank of the hospitals. However, only 50 out of the 1956 hospitals obtained from U.S. News were ranked. We considered the unranked hospitals to be at the bottom of the list. We then took the median of the unranked hospitals (ie, 1053) and considered this to be the rank of the unranked hospitals. Also, in order to account for local trends, we performed our analysis at both the national and state levels. Health care is regulated at both the state and federal levels. These regulations, along with demographics and population health, create localized trends in health care.
The results of our analysis consist of a description of general statistics about the different types of insurance and a state-wise analysis of the consumer satisfaction insurance plans. Then we report on correlations between insurances’ consumer satisfaction score and the average patient review scores of providers that accept those insurances. We report similar correlations between insurances’ overall NCQA consumer satisfaction score and then average number of referrals per provider, ratio of Castle Connolly providers, average affiliated hospital scores of providers, and relative cost of providers with respect to area. Last, we break down the providers according to their specialties and describe correlations between the average patient review scores and treatment insurance scores for condition-specialty combinations.
We first analyzed general statistics about the various insurance plans at the national level. We calculated the average overall consumer satisfaction scores of the insurance plans (see corresponding row in
General statistics about different types of health insurance plans.
Insurance plan type | Average patient review score ( |
Average consumer satisfaction insurance score ( |
Private PPO | 82.03 (<.001) | 79.75 (.384) |
Private HMO | 82.54 (<.001) | 81.63 (<.001) |
Medicaid | 82.78 (<.001) | 77.52 (<.001) |
Medicare PPO | 82.39 (<.001) | 76.71 (.263) |
Medicare HMO | 81.55 (<.001) | 76.9 (.123) |
Heat map showing average consumer satisfaction insurance scores of different plans.
Heat map showing number of health care providers per 1000 people in each state.
Heat map showing the number of health insurance plans evaluated by NCQA per state.
To estimate significance between values in the same row of
We also computed the average consumer satisfaction insurance scores for each state. The heat map in
Similarly, we computed the number of health care providers per 1000 people for each state. As shown in
Finally, we counted the number of insurance plans evaluated by NCQA per state. The heat map in
We computed the Pearson correlation of average patient review scores of providers that accept a particular insurance plan and that insurance plan’s NCQA scores. We found that there is a moderate positive correlation between these attributes (specifically .376).
Correlation between average patient review scores and consumer satisfaction insurance scores.
State | Correlation | |
Overall | .376 | |
New York | .869 | |
Texas | .794 | |
Illinois | .738 | |
Pennsylvania | .696 | |
California | .647 | |
Ohio | .549 | |
Florida | .457 |
Next, we report correlations between average referrals per provider for insurances and those insurances’ NCQA scores. Our analysis showed that there is a positive but very low correlation (specifically .031) between these two attributes. Hence, referral frequency of providers is negligibly correlated to consumer satisfaction insurance scores.
We then examined correlations between average patient review scores for specialist providers and the NCQA treatment insurance scores for these specialties. For this we used the individual treatment scores obtained from NCQA for the various conditions described in
Conditions and associated specialties ranked by correlation between NCQA scores and average patient review scores.
Condition from NCQA | Corresponding member specialties | Correlation of treatment insurance score with average patient review score |
Women’s health | Obstetrics and Gynecology, Gynecology Oncology | .135 |
Mental and behavioral health | Counselor, Psychoanalyst, Clinical Neuropsychologist, Psychologist, Psychoanalysis, Marriage and Family Therapist | .112 |
Cancer screening | Pediatric Oncology, Oncology, Hematology & Oncology, Radiation Oncology | .112 |
Heart disease | Cardiologist, Cardiac Rehabilitation, Cardiology Technician, Cardiovascular Diseases | -.002 |
Children and adolescent health | Pediatrics, Neonatal Pediatrics, Pediatrics Critical Care | -.083 |
Diabetes | Diabetes Educator, Endocrinology, Diabetes and Metabolism | -.259 |
Correlation between average patient review scores and consumer satisfaction insurance scores (overall) (correlation coefficient=.376,
Correlation between average referrals per provider and consumer satisfaction insurance scores (correlation coefficient=.031,
Correlation between ratio of Castle Connolly providers and consumer satisfaction insurance scores (correlation coefficient=.183,
Correlation between ranks of affiliated hospitals and consumer satisfaction insurance scores (correlation coefficient=.108,
Correlation between relative cost of providers with respect to area and consumer satisfaction insurance scores (correlation coefficient=.266,
Our analysis shows that there are several provider attributes that are correlated to insurance quality attributes. We showed that patient review scores for providers are correlated to consumer satisfaction insurance scores. This is expected given that patients who are happy with the care they receive from their providers are more likely to also be happy with their overall insurance plan. For example, if a patient has complaints about the billing at a provider’s office, this patient will likely be unhappy with the insurance company who did not help cover or settle the bill.
On the other hand, our results showed negligible correlation between average referrals per provider and consumer satisfaction insurance scores. This is not surprising, as there is no convincing evidence that a higher number of referrals is connected to better skills for a provider or to better relationship with patients. Similarly, we demonstrated that there is a negligible correlation between the ratio of Castle Connolly providers and the consumer satisfaction insurance scores.
The case between rank of affiliated hospitals and consumer satisfaction insurance scores was similar. However, we found a weak positive correlation between the relative cost of providers with respect to their geographic area and consumer satisfaction insurance scores. This may be explained by the fact that providers with satisfied patients may increase their prices. Of course, the charged prices are not so important, as Medicare and Medicaid generally have fixed compensations per procedure.
Our results on the lack of correlation of patient reviews score and treatment quality metrics for various conditions may indicate that patients who are satisfied with their provider may not necessarily have better health outcomes, as studies have shown that patients often rate their providers based on non‒outcome-related attributes such as wait and visit times. For instance, research has shown that the average satisfaction score for wait times of 0-15 minutes was 94.3 on a 100-point scale [
Our findings can be used to help consumers make informed choices about their insurance plans. Health insurance marketplaces may find patient review scores for providers of each insurance plan to be a useful addition to other insurance plan metrics. Alternatively, consumers can use this information in their own research to identify potential insurance plans based on the review scores of providers on review sites such as Vitals and Healthgrades.
Further, insurers may use our results to better understand the relationship between their patients’ satisfaction and their network of providers. For example, although it is not clear if there is a cause-effect relationship, our results indicate that hiring a provider with high patient review scores may contribute more to the overall consumer satisfaction insurance plan rating than hiring a provider who has been receiving many referrals from their colleagues. Further, our results indicate that more expensive providers are correlated with higher plan satisfaction, which seems to be at odds with the providers’ “tier-ing” approach of insurers, who try to encourage patients to visit the cheaper providers.
Health care providers may also use our results to decide which insurance plans to accept. As noted above, a patient whose bill was not covered by an insurance company may complain about the billing at the provider’s office on a provider review site, leading to a lower overall patient review score. A provider wishing to maintain a favorable score may thus choose to avoid accepting insurance plans with low consumer satisfaction scores.
One of our biggest limitations is that not all of the data we obtained are complete. For example, a majority of the providers have zero reviews; this is likely due to the fact that only 4% of Internet users post online reviews for providers, and previous work has shown that most providers have zero reviews [
Our data-driven analysis led to several interesting findings. Higher consumer satisfaction insurance scores are correlated with their providers having better patient review scores. There also seems to be a correlation between cost of medical care and insurance ratings. However, there was negligible correlation between other quantitative attributes such as number of referrals per provider, ratio of Castle Connolly award recipients, affiliated hospitals scores, and health insurance ratings. These findings may provide new insights into what attributes should be adopted by insurance marketplaces and search portals to empower patients in a patient-centered setting.
Centers for Medicare & Medicaid Services
Healthcare Common Procedure Coding System
Health Maintenance Organization
National Committee for Quality Assurance
National Provider Identifier
Preferred Provider Organization
The datasets on which this research relies are publicly available from [
This project was partially supported by National Science Foundation grants IIS-1216007, IIS-1447826, and IIP-1448848. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
PS built crawlers to collect data about health insurance plans from ncqa.org and created the insurance mappings, carried out the analysis, and wrote the manuscript. RR built crawlers for collecting the provider data from multiple sources and handled the integration between these sources. VH conceived the study, researched the sources, and provided coordination and guidance in the experiments and writing of the manuscript.
None declared.