Rare Adverse Medical Events in VA Inpatient Care: Reliability Limits to Using Patient Safety Indicators as Performance Measures

Abstract

Objective. To assess Agency for Healthcare Research and Quality's Patient Safety Indicators (PSIs) as performance measures using Veterans Administration hospitalization data. Data Sources Study Setting. Nine years (1997Nine years ( -2005 of all Veterans Health Administration (VA) administrative hospital discharge data. Study Design. Retrospective analysis using diagnoses and procedures to derive annual rates and standard errors for 13 PSIs. Data Collection/Extraction Methods. For either hospitals or hospital networks (Veterans Integrated Service Networks [VISNs]), we calculated the percentages whose PSI rates were consistently high or low across years, as well as 1-year lagged correlations, for each PSI. We related our findings to the average annual number of adverse events that each PSI represents. We also assessed time trends for the entire VA, by VISN, and by hospital. Principal Findings. PSI rates are more stable for VISNs than for individual hospitals, but only for those PSIs that reflect the most frequent adverse events. Only the most frequent PSIs yield significant time trends, and only for larger systems. Conclusions. Because they are so rare, PSIs are not reliable performance measures to compare individual hospitals. The most frequent PSIs are more stable when applied to hospital networks, but needing large patient samples nullifies their potential value to managers seeking to improve quality locally or to patients seeking optimal care. Key Words. Patient Safety, veterans, administrative data, reliability Patient safety has become an issue of focal interest in the evaluation of health care quality (Kohn, Corrigan, and Donaldson 1999). Recently the federal Agency for Healthcare Research and Quality (AHRQ) has developed a comprehensive set of standardized patient safety measures, derived from hospital discharge data, which potentially might be used to compare performance r Health Research and Educational Trust

‘Global Trigger Tool’ Shows That Adverse Events In Hospitals May Be Ten Times Greater Than Previously Measured

“…16 This concern has prompted the search for more direct approaches to measuring adverse events. Such approaches generally rely on more exhaustive physician or nurse reviews of a patient's complete medical record or chart.…”

Reducing Hospital Errors: Interventions that Build Safety Culture

“…The search for more accurate measures of the scope of the problem continues (18). Administrative data such as the Agency for Healthcare Research and Quality's patient safety indicators (91) provide another source of data intended to fuel change; however, some evidence suggests that they do not predict individual hospital performance (171).…”

Detecting adverse events in surgery: comparing events detected by the Veterans Health Administration Surgical Quality Improvement Program and the Patient Safety Indicators

“…Previous studies have cited coding issues as a significant factor limiting the use of the PSIs for public reporting and performance measurement. 22,39 We expected that the increased attention to coding postoperative complications might improve the PPV of some of the PSIs as compared to our previous results. The improvement in PPV for PE/DVT from 22% to 31% may reflect improvements in the documentation or coding of PE and DVT events, a focus of many healthcare systems, including the VA, following the introduction of the Surgical Care Improvement Program (SCIP) in 2005.…”

“…[18][19][20] Studies have demonstrated good predictive and construct validity for several of the surgical PSIs. [21][22][23] Additionally, some of the surgical PSIs were recently endorsed by the National Quality Forum (NQF) for quality reporting and hospital performance measurement. 24 The Centers for Medicare and Medicaid Services (CMS) publicly report private sector PSI rates on the Hospital Compare website.…”