Exploring the Behavior of Hospital Composite Performance Measures

Abstract

Background-Composite scores that combine several performance measures into a single ranking are becoming the accepted metric for assessing hospital performance. In particular, the Centers for Medicare & Medicaid Services Hospital Quality Incentive Demonstration (HQID) project bases financial rewards and penalties on these scores. Although the HQID composite calculation is straightforward and easily understood, its method of combining process and outcome measures has not been validated. Methods and Results-Using data on 530 hospitals from the Society of Thoracic Surgeons National Cardiac Database, we replicated the HQID methodology with 6 nationally endorsed performance measures (5 process measures plus survival) for coronary artery bypass surgery. Composite scores were essentially determined by process measure performance alone; the survival component explained only 4% of the composite score's total variance. This result persisted even when the survival component was allowed a 5-fold greater weighting in the composite summary. The popular "all-or-none" measurement approach was also dominated by the process component. Substantial disagreement was found among hospital rankings when several alternative methods were used; up to 60% of hospitals eligible for the top financial reward under HQID would change designation depending on the composite methodology used. The application of a simple statistical adjustment (standardization) to each method would provide more consistent results and a more balanced assessment of performance based on both process and outcomes. Conclusions-Existing methods used to create composite performance measures have remarkably different weighting of process versus outcomes metrics and lead to highly divergent provider rankings. Simple alternative methods can create more balanced process-outcome performance assessments.

ACCF/AHA 2010 Position Statement on Composite Measures for Healthcare Performance Assessment

“…2 Although statistical power and precision are potentially enhanced by incorporating process measures into assessments of providers, small sample sizes and ceiling effects (performance near 100%) are challenges for some process measures. 10 To date, empirical studies have found only a modest association between nationally reported cardiovascular process measures and outcomes, including risk-adjusted mortality. [11][12][13] Thus, a composite measure that has high accuracy for measuring adherence to evidence-based care practices may have limited accuracy for predicting an individual provider's outcomes.…”

Cardiac Report Cards

“…For these reasons, it is imperative that cardiovascular practitioners learn about the research being conducted on report cards and their methodologies. This issue of Circulation contains important new contributions from Stukenborg and colleagues 4 from the University of Virginia on the topic of improved administrative database comorbidity coding for acute myocardial infarction report cards, whereas O'Brien and colleagues 5 from Duke University address the topic of composite measures for coronary artery bypass graft surgery report cards. In this editorial, we discuss how these 2 new studies on cardiac report card methodology advance the field.…”

“…5 The Hospital Quality Incentive Demonstration composite performance measure includes 5 nationally endorsed process measures and the outcome measure of death, with each measure weighed equally. However, when applied to the Society of Thoracic Surgeons' database, the mortality rate component explained only 4% of the total variance of the composite scores, because the standard deviation (degree of spread) of mortality rates was much smaller than that of the process measures included in the composite indicator.…”

“…O'Brien et al 5 suggest that by conducting standardization (ie, measuring each element on a similar scale or range), a more balanced composite measure can be created that weighs process and outcomes performance more equally. In our view, it does seem reasonable to standardize each element of a composite measure, and it is not technically difficult to do so.…”

Comparison of Composite Measure Methodologies for Rewarding Quality of Care

“…6 Hospital rankings may vary widely, depending on the approach used to aggregate measures of quality. [7][8][9] Occasionally, multiple approaches are used together in the construction of a composite score. 10 Furthermore, there are divergent findings regarding whether or not evaluations of care processes can help improve patient outcomes.…”