Incorporating natural variation into IVF clinic league tables: The Expected Rank

Abstract

BackgroundRankings based on outcome are often used to present health care provider performance. These rankings do however not reflect that part of the variation in outcome between providers is caused by natural variation, and not by any differences in quality of care. The aim of this study is to compare standard methods for ranking with a novel method that takes into account natural variation.MethodsWe used data on the number of treatment cycles and the number of pregnancies of 13 Dutch IVF clinics from 2004. We calculated the Expected Rank (ER), an estimate of the true rank of a provider, accounting for natural variation. We rescaled the ER to obtain the Percentile based on ER (PCER), that can be interpreted as the probability that a clinic is worse than a randomly selected other clinic. We also calculated a measure for rankability ρ, which is the part of variation between providers that is due to true differences (as opposed to natural variation).ResultsThe expected ranks ranged from 1.4 to 11.9 instead of the original ranks 1–13. The ER showed that some clinics performed very similar, which would be disregarded when using standard ranks. The PCER ranged from 7% to 88%. Rankability was substantial (ρ = 0.9)ConclusionThe Expected Rank provides a way to combine the attractiveness of a ranking, a single number and easy interpretation, with reliable analyses that does justice to the providers, and also allows individual comparisons.

Random variation and rankability of hospitals using outcome indicators

“…They found considerable heterogeneity, while uncertainty per clinic was small because of large numbers (median 654 cycles). This resulted in a substantial rankability with only 10% of the observed differences between the clinics attributed to chance 20. Compared with this research, rankability in our data was much lower.…”

“…Lingsma et al used rankability to assess the ranking of a small number of in-vitro fertilisation (IVF) clinics 20. They found considerable heterogeneity, while uncertainty per clinic was small because of large numbers (median 654 cycles).…”

“…Lingsma et al suggested that over 70% rankability should be fair to rank hospitals 20. Higgins et al assigned adjectives of low, moderate and high to the I 2 values of 25%, 50% and 75% 26 .…”

“…A conceptual framework can be drawn to summarise the elements of between-hospital differences (figure 1). 20 The observed differences can be divided into unexplained differences and chance. By using random effect models, chance can be corrected for, leaving patients characteristics, registration bias, quality of care and residual confounding as elements of the unexplained differences.…”

Limitations of Ranking Lists Based on Cardiac Surgery Mortality Rates

“…Lingsma and Steyerberg propose to use expected ranks. 22 These are rank statistics based on the probability that a hospital performs worse than any other hospital in the ranking list. The expected ranks incorporate the magnitude of difference and thus allow subtle differences between hospitals (eg, rank 4, 5, and 6 versus expected rank 4.1, 4.2, and 5.9).…”