Reliability of Risk-Adjusted Outcomes for Profiling Hospital Surgical Quality

Abstract

IMPORTANCE Quality improvement platforms commonly use risk-adjusted morbidity and mortality to profile hospital performance. However, given small hospital caseloads and low event rates for some procedures it is unclear whether these outcomes reliably reflect hospital performance. OBJECTIVE To determine the reliability of risk-adjusted morbidity and mortality for hospital performance profiling using clinical registry data. DESIGN Retrospective cohort study SETTING/DATA SOURCE American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), 2009 PARTICIPANTS All patients who underwent colon resection, pancreatic resection, laparoscopic gastric by pass, ventral hernia repair, abdominal aortic aneurysm (AAA) repair, and lower extremity by pass (n=55,946 patients). MAIN OUTCOMES AND MEASURES Outcomes included risk-adjusted overall morbidity, severe morbidity, and mortality. We assessed reliability (0 to 1 scale, where 0=completely unreliable and 1=perfectly reliable) for all three outcomes. We also quantified the number of hospitals meeting minimum acceptable reliability thresholds (>0.70=good reliability, >0.50=fair reliability) for each outcome. RESULTS For overall morbidity, the most common outcome studied, the average reliability depended on both the sample size (i.e., how high the hospital caseload was) and the event rate (i.e., how frequently the outcome occurred). For example, average reliability for overall morbidity was low for AAA repair (reliability 0.29; sample size of 25 cases/year and event rate of 18%). In contrast, average reliability for overall morbidity was higher for colon resection (reliability 0.61; sample size 114 cases/year and event rate of 27%). Colon resection (38% of hospitals), pancreatic resection (7% of hospitals), and laparoscopic gastric by pass (12%) were the only procedures for which any hospitals met a reliability threshold of 0.70 for overall morbidity. Because severe morbidity and mortality are less frequent outcomes, their average reliability was lower and even fewer hospitals met thresholds for minimum reliability. CONCLUSIONS AND RELEVANCE Most commonly reported outcome measures have low reliability for differentiating hospital performance. This is especially important for clinical registries that sample rather than collect 100% of cases, which can limit hospital case accrual. Eliminating sampling to achieve the highest possible caseloads, adjusting for reliability, and using advanced modeling strategies (e.g., hierarchical modeling) is necessary for clinical registries to increase their benchmarking reliability.

Preoperative identification of neurosurgery patients with a high risk of in-hospital complications: a prospective cohort of 418 consecutive elective craniotomy patients

“…10 This retrospective study could not assess the role of routine preoperative blood tests, ASA class, or any other preoperative risk score in outcome prediction. Furthermore, the reliability of predicting outcomes by using registry-based data has been questioned by others 5 and even by the authors themselves, 10 specifically since in many large registries detailed preoperative patient data are lacking. Another major problem is that the type and rate of complications in neurosurgery are variable but limited.…”

Assessing early unplanned reoperations in neurosurgery: opportunities for quality improvement

“…15 Review of morbidities and mortality has been the primary method of assessing surgical quality used by physicians, hospitals, and oversight agencies. 10,17,20 In recent years, the reoperation rate, defined as the percentage of patients undergoing a new operation, has been proposed as a quality indicator, a surrogate for surgical adverse events. 4,13,19,24 General surgery has been at the forefront of assessing reoperations.…”

Reliability of Hospital Cost-Profiles in Inpatient Surgery

“…sample size) at most hospitals and low rates of most adverse events. 2,3 As a result, an emphasis has been placed on considering reliability when selecting outcome measures. 4,5,[14][15][16] In contrast to other clinical outcome measures, we have demonstrated that episode payments are more reliable and many hospitals meet commonly accepted reliability benchmarks, even for less common procedures such as pancreatectomy.…”

“…2,3 While the reliability of outcome measures such as 30-day mortality and complications for surgical procedures have been investigated, the reliability of total episode payments, a measure that is associated with each episode of care, has not yet been explored. [3][4][5][6] A better understanding of surgical episode payment reliability would provide additional information on how confidently this proposed quality metric could be used to compare providers, establishing an additional metric for assessing episode efficiency. This has important implications in helping Accountable Care Organizations (ACO) and hospitals interested in bundled payments understand potential liabilities and opportunities for improvement under these newer payment strategies.…”