Simpson's paradox: how performance measurement can fail even with perfect risk adjustment

The multiple aims of pay-for-performance and the risk of unintended consequences

“…Does this mean the risk adjustment should be changed in HRRP? Second, given that the risk adjustment will never be perfect,18 what other approaches are available to maximise the potential of pay-for-performance to benefit patients, while reducing the risk of unintended consequences?…”

Better-than-average and worse-than-average hospitals may not significantly differ from average hospitals: an analysis of Medicare Hospital Compare ratings

“…For example, providers flagged as outliers using a funnel plot would not necessarily significantly differ from a given non-outlier provider,19 and providers that exceed a predetermined performance threshold (eg, 75th centile) would not necessarily significantly differ from those that do not 20. Although the publicly available Medicare Hospital Compare estimates were risk-adjusted, it is important to note that our findings apply to both risk-adjusted and non-adjusted performance estimates; information about the strengths and limitations of risk adjustment strategies is available elsewhere 21 22…”

What is a performance outlier?

“…In all these examples, even with perfect patient-level risk adjustment, comparisons among providers may be biased and inaccurate unless differences in the relative distributions of higher and lower risk cases are properly accounted for,4 23 a profiling analogue of Simpson's paradox 24 25. The impact of this phenomenon is not uniform.…”