Association of Electronic Health Record Vendors With Hospital Financial and Quality Performance: Retrospective Data Analysis

Background: Electronic health records (EHRs) are a central feature of care delivery in acute care hospitals; however, the financial and quality outcomes associated with system performance remain unclear. Objective: In this study, we aimed to evaluate the association between the top 3 EHR vendors and measures of hospital financial and quality performance. Methods: This study evaluated 2667 hospitals with Cerner, Epic, or Meditech as their primary EHR and considered their performance with regard to net income, Hospital Value–Based Purchasing Total Performance Score (TPS), and the unweighted subdomains of efficiency and cost reduction; clinical care; patientand caregiver-centered experience; and patient safety. We hypothesized that there would be a difference among the 3 vendors for each measure. Results: None of the EHR systems were associated with a statistically significant financial relationship in our study. Epic was positively associated with TPS outcomes (R=23.6%; β=.0159, SE 0.0079; P=.04) and higher patient perceptions of quality (R=29.3%; β=.0292, SE 0.0099; P=.003) but was negatively associated with patient safety quality scores (R=24.3%; β=−.0221, SE 0.0102; P=.03). Cerner and Epic were positively associated with improved efficiency (R=31.9%; Cerner: β=.0330, SE 0.0135, P=.01; Epic: β=.0465, SE 0.0133, P<.001). Finally, all 3 vendors were associated with positive performance in the clinical care domain (Epic: β=.0388, SE 0.0122, P=.002; Cerner: β=.0283, SE 0.0124, P=.02; Meditech: β=.0273, SE 0.0123, P=.03) but with low explanatory power (R=4.2%). Conclusions: The results of this study provide evidence of a difference in clinical outcome performance among the top 3 EHR vendors and may serve as supportive evidence for health care leaders to target future capital investments to improve health care delivery. (J Med Internet Res 2021;23(4):e23961) doi: 10.2196/23961


Background
In the first part of the 20th century, health care predominantly revolved around a single health care provider, diagnosing and treating patients within the confines of their office or in the patient's home, and patient medical histories were recorded on paper. However, as the US health care industry has progressed and modernized, the proliferation of information technology has accelerated. In 2013, US health information technology investment totaled US $2.8 billion; however, by 2017, it had reached a staggering US $7.1 billion [1]. However, some reports indicate that hospitals have struggled to remain profitable during this same time frame [2]. Many health care executives are questioning the return on investment in hospital technology and wondering whether their capital outlay will result in improved financial outcomes [3].
Numerous studies have noted the high expense of care delivery yet low health care information technology proliferation in the United States. Citing the advancement of similar technology in other developed countries, researchers have indicated that through increased investment in health information technology, the United States can lower overall health care spending and simultaneously improve quality of care and patient outcomes [4,5]. As a result, the federal government has been heavily involved in the modernization of health information technology. As early as 2004, President George W Bush called for computerized health records in his State of the Union address and offered a strategy to provide Americans access to electronic health records (EHRs); unfortunately, he did not earn sufficient funding to change provider behavior [6]. In 2009, President Barack Obama signed the Health Information Technology for Economic and Clinical Health (HITECH) Act as part of the American Recovery and Reinvestment Act (ARRA) and originally set aside US $27 billion for an incentive program that encouraged hospitals and providers to adopt EHR systems [7,8]. This legislation prompted the adoption of EHRs in 2014 and established time frames for mandated EHR adoption. Furthermore, with the passage of the Medicare Access and CHIP Reauthorization Act, Medicare introduced the Medicare EHR Incentive Program, and the Merit-Based Incentive Payment System to advance the meaningful use of EHRs. As a result, by 2015, 96% of hospitals and 87% of physician practices implemented EHRs [9]. By mid-2016, the total federal government investment in EHR rose to US $35 billion and continued to rise [10].
One could argue that the proliferation of health care technology spending did not occur organically. The hospital and health care industry shifted to the use of EHR systems primarily because of the financial incentives incorporated within the ARRA and HITECH legislation [11]. HITECH provided eligible professionals who demonstrated the meaningful use of an EHR qualified for payments of US $18,000 in the first year; US $12,000 for the second year; US $8000 for the third year; US $4000 for the fourth year; and US $2000 for the fifth year [12]. An eligible professional was generally considered to be a physician. After 2015, physicians who failed to meaningfully use EHRs were subject to reductions in Medicare and Medicaid reimbursement. Meaningful use of an EHR includes 3 components: (1) the EHR must be certified and include e-prescribing capabilities; (2) the technology must provide for the electronic exchange of personal health information with other EHR systems (interoperability); and (3) the system must produce reports utilizing various clinical and quality metrics.
Beginning in 2011, incentive payments were also available for eligible hospitals that showed meaningful use of EHR and that submitted quality metrics based on criteria identified by the US Department of Health and Human Services. Incentive amounts were phased out in 2015 for hospitals that had not implemented a meaningful EHR. In addition, incentive payments were not available for hospitals that were not meaningful EHR users [13]. Despite the robust startup incentives offered by the federal government and continuing support provided via increased Medicare reimbursement, questions remain if the investment in health care information technology is sustainable. The adoption of a comprehensive EHR system can surpass several billion dollars for a large health care system [14]. The sustainability of these systems can approach several hundred million dollars annually, and numerous health care systems report significant implementation and sustainment cost overages [15,16]. Furthermore, despite the prevalent adoption of EHR systems since the passage of the HITECH Act, sharing of health care data and interoperability of information technology remains to be elusive. There remains to be little financial incentive to share and use data to reduce costs or improve the quality of care [17].
Thus, we seek to assess the association between investment in information technology and the dominant EHR platforms each have on the financial and quality outcomes of hospitals in the United States. We hypothesize that there is a difference in outcomes among the 3 vendors for each measure. Although extensive research has focused on the perceived and actual benefits of information technology in health care, this is an area of research that has not been fully evaluated. The body of literature predating the passage of the HITECH and Affordable Care Acts is fairly robust; however, with the passage of these legislative acts and the rate of change in technology, many of these studies are now outdated, particularly with respect to the direct impact of specific EHR vendors [18][19][20].

Literature Review
Two recent studies examined the relationships between health information technology capital expenditure and both financial and quality outcomes and aligned very closely with our work. Wang et al [21] examined the impact of investment in health information technology on hospital financial performance and productivity. In a later study, Wang and Gibbs [22] offered a framework to compare the performance of EHR systems. We intend to build on these studies in a few key areas [21,22].
First, in the 2018 and 2019 studies, the key financial outcome variable examined was return on assets (ROA). Net revenue per staffed bed was also considered in the 2018 study. Broadly speaking, financial performance can be assessed in 4 main areas: (1) profitability or return on investment, (2) liquidity, (3) leverage, and (4) operating efficiency. Within each category, there are several variables to consider. Although ROA and revenue per bed are important, we believe that greater operational clarity can be achieved with a specific focus on the income statement and net income.
Second, quality was evaluated in the 2019 study via the Hospital Value-Based Purchasing (HVBP) Total Performance Score (TPS). These quality measures are all important to the field; however, we view the level of utilization of beds as a suboptimal measure of quality that is not well supported in the literature. The number of times a bed turns over speaks more to the volume of services demanded but offers little insight with respect to the quality of those services. We intend to focus more on the dimensions of HVBP and its subdomains. The contributory factors that are evaluated to produce the TPS include patient perceptions of care and measures of patient safety, process of clinical care delivery, and efficiency and cost reduction.
Third, both the 2018 and 2019 studies included important market characteristic variables as controls. These included hospital size, market concentration index (MCI), payer mix (Medicare and Medicaid; ie, the percentage of revenue coming from each of those programs), uncompensated care cost, ownership (governmental, proprietary, and nonprofit), teaching status, geographic classification, and year fixed effects. We suggest that additional variables may be insightful, as several other factors have been shown to influence both financial and quality outcomes [23][24][25]. Additional control variables worth considering are the level of outpatient services rendered, urban versus rural location, average length of stay, case mix index, wage index, sole community provider status, system membership, and geographic region [26]. These variables can further clarify the strength of the association between the independent variable of interest and our targeted dependent variables, while also serving to diminish any possible omitted variable bias.
Fourth, the 2019 study considers various EHR systems but does not clearly identify which vendors perform better or worse based on the available data. To the authors' credit, they did not want the research to be construed as the promotion of a particular vendor. However, in our view, the health care industry is in an era of evidence-based medicine and management. Thus, we believe that it is appropriate to identify the system in question. This more transparent approach, coupled with more specific outcome data that clearly identifies practitioner actionable evidence, should provide greater practical insight and facilitate improved organizational decision making.
In the following sections, we integrate and broaden the investigation of previous research efforts and evaluate the impact of 3 of the largest EHR providers' performance on measures of finance and quality. Quite simply, we would like to determine which EHR system performs the best and seek to provide health care leaders with an additional evidentiary basis for making EHR adoption decisions. Given the variation in EHR system cost, options, ease of use, training requirements, and on-site and follow-up support, we recognize that this can be a highly complex decision. Although we hypothesize that there is a difference in performance among the 3 EHR vendors in terms of financial and quality performance, we are uncertain a priori where each system will perform the best on the evaluation measures we have selected.

Data and Sample
The data for this study were extracted from two primary sources: the Definitive Health Care database and the American Hospital Association (AHA) Annual Survey database for 2018. The Definitive Health Care database provided the dependent and independent variables of interest, in addition to most of the control variables for this study. The Definitive Health Care database compiles US hospital data sources including Medicare Cost Reports, commercial claims data, Medicare Standard Analytics Files, Centers for Medicare and Medicaid Services (CMS) Hospital Compare, and many other data elements [27]. The cost report contains provider information such as facility characteristics, utilization data, cost and charges by cost center (in total and for Medicare), Medicare settlement data, and financial statement data. The AHA Annual Survey database provided the remaining data for the geographic region control variables [28]. All variables were linked with the 2 contributing data sources based on the Medicare provider number. Data on a total of 2667 short-term acute care hospitals were accumulated for analysis. Table 1 shows the full complement of the study variables. Our study included 2 types of dependent variables drawn from the Definitive Health Care data set. The first set of data comes from the hospital income statement: net income scaled in millions of dollars. The second set of dependent variables is drawn from the hospitals' 2018 value-based purchasing scores and includes (1) the TPS, (2) patient experience score, (3) clinical process score, (4) efficiency score, and (5) the safety score. Each hospital's TPS is a weighted measure of performance based on each of the other areas listed, while each subordinate measure is an aggregation of several commonly tracked clinical and administrative criteria, as shown in Figure 1.

Measures-Dependent Variables
Under the CMS HVBP Program, Medicare makes incentive payments to hospitals based on how well they perform on each measure compared with other hospitals' performance during a baseline period and how much they improve their performance on each measure compared with the baseline reporting period [29]. All value-based purchasing variables are unweighted and based on a scale of 0 to 100, with higher scores being better, and have been validated by CMS for validity and reliability [30]. The total number of value-based purchasing participating hospitals provided a size limit to the study. A missingness map using Amelia, a program for missing data developed by Honaker et al [31], revealed approximately 1% missing from 2667 observations and 32 variables (k). The maximum proportion missing from any column was 12.5% and from any row was 10.1%; therefore, these values were conservatively imputed with the median using R Statistical Software [32]. The Herfindahl-Hirschman Index measure of market concentration was used to determine market competitiveness. It is calculated by squaring the market share of each firm competing in a market and then summing the resulting numbers Definitive Health Care Market concentration Average age of facility is calculated using the accumulated depreciation (total depreciation) and the depreciation expense (depreciation over a single period)

Measures-Independent and Control Variables
Our independent variables of interest included the top EHR systems used by hospitals in the United States (ie, Cerner, Epic, Meditech, or other), as reported in the Definitive Health Care database. These variables were included in our analysis as a dichotomous variable for each EHR system of interest ("1" if the system was used, "0" if not). Consistent with previous research, we also included several organizational-level control variables to account for other explanatory factors that could influence financial and quality outcomes. These variables included for-profit ownership status, number of beds, rural or urban geographic location, government ownership, teaching status, outpatient service mix, average length of stay, case mix, government payer percentage, wage index, sole community provider designation, system membership, MCI, occupancy rate, and geographic location by the AHA region. All analyses were performed using collinearity diagnostics. The variance inflation factor exceeded 10 in our analyses. Table 1 shows the data sources and operational definitions of the variables used in this study.

Overview
Descriptive statistics and pairwise correlations were also calculated. The distributions of most of the dependent variables (income, TPS, experience score, clinical score, and safety) were relatively normal; however, the HVBP efficiency score was skewed to the right. Box-Cox analysis of the variable suggested a negative square root transformation, but for interpretability, it was not transformed. All dependent variables were min-max scaled between 0 and 1 for easier interpretation as percentile scores (eg, income percentile). All statistical analyses were performed using the R Statistical Software [32]. In all the analyses, a two-tailed P value <.05 was considered statistically significant.    Table 3 reflects the results of our regression analyses of hospitals' utilization of the top 3 EHR vendors and the associated hospital financial performance as measured by net income. On the basis of our analysis of net income regressed on EHR vendors (R 2 =10.6%), we see no significant results for any of the vendors, when compared with facilities that fall into the "other" category. Thus, we can say, on average, and when controlling for the numerous organizational factors as controls, none of the EHRs are associated with favorable or unfavorable financial outcomes as measured by net income.   Table 3 also shows the results of our regression analyses of the association of EHRs with HVBP quality measures. On the basis of our analysis of TPS regressed on EHR vendors (R 2 =23.6%), Epic reflects only statistically significant results (β=.0159, SE 0.0079; P=.04). These results indicate that the Epic EHR is associated with a 1.6% higher performance score when compared with facilities that fall into the "other" category. Neither Meditech nor Cerner were associated with significant results. Among other findings, these results imply, on average, for-profit facilities perform 1.7% lower on the TPS measure. In addition, with each day increase in average length of stay, we observed a 2.2% decrease in TPS, and with each point increase in market concentration, we see an associated 3.8% decrease in TPS performance. However, with each point increase in the case mix index, wage index, and outpatient service mix, we observed a 2.5%, 5.2%, and 21.4% increase in TPS outcomes, respectively. Teaching hospitals are also associated with a 1.2% higher level of performance than nonteaching facilities.

Efficiency Score
In Table 4, we also show the evaluation of efficiency performance scores regressed on EHR vendors (R 2 =31.9%). Cerner (β=.0330, SE 0.0135; P=.01) and Epic (β=.0465, SE 0.0133; P<.001) were positively associated with improved efficiency quality scores approximately 3.3% higher (Cerner) and 4.7% higher (Epic) than hospitals in the "other" category. Meditech was not associated with any significant results. , and several regional variables. This implies, on average, there is a statistically significant 2.3% decrease in efficiency score with each day increase in average length of stay, a 6.9% decrease with each point increase in the case mix index, and a 24.9% decrease in efficiency is associated with each point increase in government payer percentage.  Table 4 provides the final analysis results of our evaluation of hospitals' patient experience performance scores regressed on EHR vendors. Epic was positively associated with higher patient perceptions of quality scores 2.9% higher than hospitals in the "other" category (R 2 =29.3%; β=.0292, SE 0.0099; P=.003). These results imply that, on average, the for-profit hospitals in our study scored 5.4% lower on the HVBP patient experience scores. In addition, for each additional day in the hospital, the patient experience scores decreased by 2.5%. Each point increase in market concentration and occupancy rate also reduces patient experience by 5.9% and 6.2%, respectively. Conversely, rural and teaching hospitals are associated with higher patient experience, with associated increased scores of 5.2% and 2.7%, respectively. Table 5 provides insight into our research on patient safety performance scores regressed on EHR vendors (R 2 =24.3%). Epic (β=−.0221, SE 0.0102; P=.03) was negatively associated with patient safety quality scores of 2.2% lower than hospitals in the "other" category. Meditech and Cerner scores were not associated with significant results. Table 5 also provides details regarding the significant associations between the control variables included in our study and patient safety scores. These variables include the number of hospital beds (β=−.0002, SE 0.0000; P<.001), rural status (β=.0420, SE 0.0109; P<.001), teaching (β=.0168, SE 0.0071; P=.02), outpatient service mix (β=.0965, SE 0.0330; P=.003), average length of stay (β=−.0143, SE 0.0042; P<.001), case mix (β=−.0812, SE 0.0144; P<.001), government payer percent (β=.0746, SE 0.0288; P=.009), and occupancy rate (β=−.0919, SE 0.0238; P<.001). These results indicate that patient safety is negatively impacted by 1.4% for every day increase in average length of stay and also declines by 8.1% for every point increase in the case mix index. Furthermore, with each percent increase in the hospital occupancy rate, patient safety scores declined by 9.2%. Conversely, patient safety scores were positively associated with rural and teaching hospitals by 4.2% and 1.7%, respectively. In addition, with each percentage increase in government payments, patient safety scores improved by 7.5%.

Clinical Care Performance Score
Finally, Table 5 shows the results of our evaluation of clinical care performance scores regressed on EHR vendors. All 3 vendors were associated with positive performance with Epic (β=.0388, SE 0.0122; P=.002), Cerner (β=.0283, SE 0.0124; P=.02), and Meditech (β=.0273, SE 0.0123; P=.03), reflecting positively associated higher clinical care performance scores between 2.7% (Meditech) and 3.8% (Epic) higher than hospitals in the "other" category. However, on this dependent variable, we recognize that the explanatory power of the regressors is very low (R 2 =4.2%). These results indicate that government-operated hospitals are associated with 2.6% lower clinical process scores, and for each point increase in government payer percentage, there is also an 8.3% lower score. However, for-profit, teaching, and system-owned hospitals appear to perform better on this measure by 5.1%, 1.6%, and 3.3%, respectively. Hospitals in concentrated markets also appear to perform better than those in less concentrated markets. With each point increase in market concentration, we observed an increase of 3.2%.

Principal Findings
In general, our findings were insightful regarding the performance of individual EHR vendors. We did not expect to see a clearly obvious choice of EHR vendor with respect to performance as defined by our financial and quality-focused dependent variables. To this end, we did not ascertain that there is a single EHR that outperforms all other competitors across all of our study measures.
Our findings pertaining to financial outcomes were somewhat interesting in that no single EHR demonstrated a significant and positive association with net income. In most instances, the capital allocation process is predicated on reasonable assurance that there will be some tangible return on investment over a reasonable amount of time. As EHR adoption is a major capital investment and requires a major organizational change and extensive training, some have argued that it may take the organization a few years to see its effect on financial performance. For instance, Collum et al [20] found a statistically significant improvement in the total margin 2 years after EHR adoption in hospitals. However, the authors attributed the observed effect more to HITECH Act incentive payments than operational improvements, primarily because the authors found no significant association with operating margin. Thus, these previous authors' observations, coupled with our own, continue to indicate that EHR return on investment remains an unsettled matter.
In our evaluation of TPS as a dependent variable, we note that Epic was the singular EHR with a positive and significant association with improved scoring. Given that the TPS is a composite of the other variables in our analysis, it prompted us to examine each of the subdomains' performance scores more closely. Although we did not see a positive association for Epic in improved financial performance, this EHR recorded positive and significant associations in clinical care, patient experience, and efficiency scoring. Thus, when considered together, this combination of scores across these 3 quality subdomains appears to provide a performance advantage to Epic and a positive association with TPS scoring.
However, Epic also demonstrated a statistically significant and negative association with patient safety, which was not observed in our other vendors' performance. Although we did not expect to observe a significant and negative association between any EHR vendors and patient safety scoring, upon further research on this topic, we note that our findings appear to be consistent with several previous researchers' results. Bowman [33] captures these areas of concern very well in her synthesis of 64 studies and papers highlighting numerous areas where EHRs can impose undue burdens on health care providers and introduce the possibility of errors. These include the potential for recording erroneous data entry leading to patient safety hazards, system design flaws, improper system use, inappropriate document capture, erroneous application of copy and paste functions within the medical record, rigid application of prepopulated templates, and errors related to clinical decision support systems such as alert fatigue [33]. In recent years, others have pointed to potential problems with EHRs as a vector for increased risk to patient safety with respect to incorrect use [34,35], malfunctions [36,37], interoperability or system interaction [38], and health information technology blackouts or downtime [36,39]. On the basis of our findings, we can reasonably assume that many of these issues persist.
Our research group discussed the possible reasons for these results. In many ways, our results are supported by independent research. The top vendors that we studied are often highlighted in the KLAS, LLC Research for best in class, most user friendly, and holding buyers' attention [40]. It is a user-friendly variable that could contribute to the software's success in efficiency measures. It is possible that all 3 vendors are equally capable of achieving similar efficiency scores, but the fact that users are more familiar with their function and more willing to explore beyond the basic user training that renders Epic more effective in the areas of TPS. One could reasonably assume that the number of years since the hospital adopted the EHR system, and also the stage of adoption, might have had an effect. Hospitals that have adopted the EHR over several years may be more efficient than hospitals that only recently adopted or changed their EHR system. This could have an impact on levels of customer service, capability to integrate modules together, onboarding processes relating to the EHR, and the organization's capacity to facilitate initial and ongoing training.
Finally, another factor contributing to the success of one vendor over the others could be ownership. Epic and Meditech have proudly and defiantly maintained their private status. This factor could make a vendor more agile in its software development life cycle, enabling them to customize to order or correct flaws rapidly.

Practice Implications
Those involved in purchasing decisions surrounding EHR should carefully consider areas of focus in the facility, capital expenditure cycles, strategic direction, and willingness of the organization to change EHR vendors. The significant decision to switch EHR vendors is a complex process, and many factors need to be aligned to set up the organization for success. Our research may provide an evidentiary basis for vendor selection. For example, an organization that struggles with improving clinical care performance might consider Epic or Meditech with an understanding that there are numerous other factors that might influence outcomes. A similar choice might be considered for facilities desiring to improve patient experience. However, if patient experience or efficiency-in terms of Medicare Spending Per Beneficiary-is an area of weakness, Epic might be a preferred choice. Ultimately, an organization using a vendor other than these 3 might look at the features that these vendors offer that their vendor does not. Is there something their current vendor could offer and increase measures of efficiency? Regrettably, our research does not extend to the module level of the EHR, so we cannot make any recommendations in that regard.

Limitations and Recommendations for Future Research
Our study had several limitations. First, this is a single year of data drawn from the 2018 data pertaining to performance within only short-term acute care HVBP participating facilities. Future studies should consider examining the growth or decline of EHR influence on these outcomes over time. Furthermore, as a single-year study, our analysis also does not account for any changes in EHR systems during the delay between baseline and performance reporting periods from which HVBP scores were determined, nor do we include the length of time the EHR has been in place within the hospitals studied. A more in-depth paired analysis could be considered to match the EHR system with the exact time frame of performance. Additional financial and quality outcome variables might also be considered, which could broaden the study unit of observations beyond the HVBP constraint. Finally, as more granular data becomes available pertaining to the specific modules in use at the hospital level, future studies might examine how specific module use is associated with specific clinical outcomes.
Beyond the extensive implementation of EHRs, health care providers, hospitals, and health care facilities invest heavily in other forms of information technology to provide and enhance care delivery. As the industry progresses toward value-based care, organizations are increasingly investing in imaging, telehealth, precision medicine, artificial intelligence, cloud-based computing or data storage, consumer-facing technologies, and disease management technologies. Furthermore, in the days since the start of the COVID-19 pandemic, telemedicine has been showcased as an indispensable capability of the EHR. Another aspect that should be evaluated in the future is the telemedicine capabilities of these vendors. Future research should carefully examine care delivered through this modality and compare the outcomes across the top vendors.

Conclusions
The return on investment and outcomes associated with EHRs have been a topic of intense focus and debate over the past 2 decades. Up to this point, a research gap has persisted pertaining to the study and transparent disclosure of comparative studies of major EHR vendors. In our analysis of the big 3 vendors-Epic, Cerner, and Meditech-we endeavored to fill that gap. Yet, we can see that clearly answering which system performs the best is complex. The implementation of any of these products can take years, and success is not guaranteed. However, our findings may provide some clarity to health care leaders seeking to develop an evidence base to support future capital investment in EHR systems. If an organization is already considering a switch to a new EHR vendor, the organization can devote sufficient funding for such an undertaking, and if the leadership is willing to lead such a large organizational change, then our study may provide some points of clarity pertaining to the big 3 vendors that might be apt for consideration.