Ranking Hospitals Based on Preventable Hospital Death Rates: A Systematic Review With Implications for Both Direct Measurement and Indirect Measurement Through Standardized Mortality Rates

Abstract

The use of standardized mortality rates (SMRs) to profile hospitals presumes differences in preventable deaths, and at least one health system has suggested measuring preventable death rates of hospitals for comparison across time or in league tables. The influence of reliability on the optimal review number per case note or hospital for such a program has not been explored. Estimates for preventable death rates using implicit case note reviews by clinicians are quite low, suggesting that SMRs will not work well to rank hospitals, and any misspecification of the risk‐adjustment models will produce a high risk of mislabelling outliers. Most studies achieve only fair to moderate reliability of the direct assessment of whether a death is preventable, and thus it is likely that substantial numbers of reviews of deaths would be required to distinguish preventable from nonpreventable deaths as part of learning from individual cases, or for profiling hospitals. Furthermore, population‐ and hospital system–specific data on the variation in preventable deaths or adverse events across the hospitals and providers to be compared are required in order to design a measurement procedure and the number of reviews needed to distinguish between the patients or hospitals. Context There is interest in monitoring avoidable or preventable deaths measured directly or indirectly through standardized mortality rates (SMRs). While there have been numerous studies in recent years on adverse events, including preventable deaths, using implicit case note reviews by clinicians, no systematic reviews have aimed to summarize the estimates or the variations in methodologies used to derive these estimates. We reviewed studies that use implicit case note reviews to estimate the range of preventable death rates observed, the measurement characteristics of those estimates, and the measurement procedures used to generate them. We comment on the implications for monitoring SMRs and illustrate a way to calculate the number of reviews needed to establish a reliable estimate of the preventability of one death or the hospital preventable death rate. Methods We conducted a systematic review of the literature supplemented by a reanalysis of authors’ previously published and unpublished data and measurement design calculations. We conducted initial searches in PubMed, MEDLINE (OvidSP), and ISI Web of Knowledge in June 2010 and updated them in June 2012 and December 2017. Eligibility criteria included studies of hospital‐wide admissions from general and acute medical wards where preventable death rates are provided or can be estimated and that can provide interobserver variations. Findings Twenty‐three studies were included from 1985 to 2017. Recent larger studies suggest consistently low rates of preventable deaths (interquartile range of 3.0%‐6.0% since 2008). Reliability of a single review for distinguishing between individual cases with regard to the preventability of death had a Kappa statistic of 0.10‐0.50 for deaths and 0.21‐0.76 for adverse events. ...

Rate of Preventable Mortality in Hospitalized Patients: a Systematic Review and Meta-analysis

“…Another well-described limitation with physician review is poor inter-rater reliability. 32 Indeed, the inter-rater reliability for preventable death of included studies was generally in the fairmoderate range.…”

“…Overall hospital mortality rates and disease-specific mortality rates continue to be reported in many countries in Europe and the USA. 32,33 In the USA, overall hospital mortality rates are reported by the Veterans Health Administration and disease and procedure-specific mortality rates are used by the Centers for Medicare and Medicaid Services (CMS). Disease-specific mortality rates are also used to determine hospital reimbursement as part of CMS' Hospital Value-Based Purchasing Program.…”

Increasing specialist intensity at weekends to improve outcomes for patients undergoing emergency hospital admission: the HiSLAC two-phase mixed-methods study

“…127 Consensus-based joint reviews create an illusory improvement in reliability. 128 Averaging across multiple reviews improves reliability; 129 the approach we have adopted, with 4000 case records in total and 4763 usable reviews covering 20 trusts in two epochs, has led to one of the largest reviews undertaken, and distinguishes reliably between trusts (Spearman-Brown coefficients 0.8-0.9). Our conclusions are enhanced by triangulation with qualitative research (involving interviews, and observations employing elements of an ethnographic approach) and employing a difference-in-difference analysis that minimises confounding from variation in case mix.…”

Analysis of structure indicators influencing 3-h and 6-h compliance with the surviving sepsis campaign guidelines in China: a systematic review

“…The higher that the predicted mortality rate is, the higher that the proportion of patients admitted to the ICU with severe diseases is. The lower that the standardized mortality ratio is, the higher that the diagnosis and treatment level in the ICU is [ 15 – 17 ]. The combination of the above two indicators can fully reflect the medical level of an ICU.…”