Quantifying the Potential Bias when Directly Comparing Standardised Mortality Ratios for In-Unit Neonatal Mortality

Abstract

IntroductionThe Standardised Mortality Ratio (SMR) is increasingly used to compare the performance of different healthcare providers. However, it has long been known that differences in the populations of the providers can cause biased results when directly comparing two SMRs. This is potentially a particular problem in neonatal medicine where units provide different levels of care.MethodsUsing data from The Neonatal Survey (TNS), babies born at 24 to 31 weeks gestational age from 2002 to 2011 and admitted to one of 11 UK neonatal units were identified. Risk-adjusted SMRs were calculated for each unit using a previously published model to estimate the expected number of deaths. The model parameters were then re-estimated based on data from each individual neonatal unit (“reference” unit) and these then applied to each of the other units to estimate the number of deaths each unit would have observed if they had the same underlying mortality rates as each of the “reference” hospitals. The ratios of the SMRs were then calculated under the assumption of identical risk-specific probabilities of death.Results7243 babies were included in all analyses. When comparing between Network Neonatal Units (Level 3) the ratio of SMRs ranged from 0.92 to 1.00 and for the comparisons within Local Neonatal Units (Level 2) ranged from 0.79 to 1.56. However when comparing between neonatal units providing different levels of care ratios up to 1.68 were observed.ConclusionsIf the populations of healthcare providers differ considerably then it is likely that bias will be an issue when directly comparing SMRs. In neonatal care, the comparison of Network Neonatal Units is likely to be useful but caution is required when comparing Local Neonatal Units or between units of different types. Tools to quantify the likely bias are required.

Differences in case-mix can influence the comparison of standardised mortality ratios even with optimal risk adjustment: an analysis of data from paediatric intensive care

“…Variation in the values of the SMRs reduced as the size of the PICU increased, similar to the pattern seen in acute neonatal care 26. The variation was less between the largest units, partly because the large units had patient populations similar to each other but also because their case-mix was similar to the overall population: both of these factors reduce the variation in the observed SMRs.…”

“…We have previously reported the potential differences in the value of SMR for neonatal units in the East Midlands and Yorkshire regions of England 26. It was shown that, when applying the risk-specific mortality of one neonatal unit to another and the ratio of their SMRs calculated, that these ratios ranged from 0.79 to 1.68: that is, the value of the SMR for one unit was 68% greater than that of another even when their risk-specific probabilities of death were identical.…”

“…While it is useful in quantifying this difference in case-mix, the size of the bias that can occur when comparing two SMRs is also dependent on the size of the difference between the risk-specific probabilities of death observed in the reference population and the probabilities for the two providers 26. This would also need to be taken into account in any tool to predict which comparisons would likely be misleading.…”

Changes in Survival Rate for Very-Low-Birth-Weight Infants in Korea: Comparison with Other Countries

“…These results are similar to those of a previous study by Kim et al ( 17 ), who reported that improvements in survival rates of premature infants in Korea were dramatic during the early 1990s, and studies from the United States and Canada reporting that continued improvements in VLBWI survival rates were no longer seen after the 2000s ( 3 4 26 ). Actually, confirming the age-based survival rates according to previously published reports without collecting continuous treatment outcome data can create a significant amount of errors ( 3 13 14 27 28 29 ). Particularly in Korea, nationwide surveys on premature infant survival rates prior to the 2000s did not exist, and mostly treatment outcomes from single institutes (including four multi-center studies) of varying sizes and levels had been reported with some overlapping survey periods.…”

The aggregation paradox for statistical rankings and nonparametric tests

“…Further developing the Paradox for non-parametric settings, Haunsperger [18] discovers a computable criterion by which the consistency (upon aggregation) of an ordinal data set (when KW-tested) can be characterized. Bargagliotti [34] finds that Bhapkar's V test [35] and the Wilcoxon-Mann-Whitney (WMW) test [Wilcoxon 1945, Mann andWhitney 1947]-two other central non-parametric tests-are also not generally consistent upon aggregation. Hao and Houser [38] state, "Both robust and simple to implement, [the WMW test] has gained exceptional popularity among empirical scientists, even social scientists.…”