Ranking hospital performance based on individual indicators: can we increase reliability by creating composite indicators?

Austin, Peter C.; Ceyisakar, Iris; Lingsma, Hester F.; Mheen, Perla J Marang-van de

doi:10.1186/s12874-019-0769-x

Cited by 18 publications

(22 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, low rankability is defined as ,50% and no ranking attempt should be made when rankability is low. 11,13 Besides our definition of high rankability as above 75%, 11 it has been suggested that above 70% is fair to rank hospitals, 14 meaning QI 6 was the only fair indicator.…”

Section: Discussionmentioning

confidence: 92%

Effect of Case-Mix and Random Variation on Breast Cancer Care Quality Indicators and Their Rankability

et al. 2020

Self Cite

View full text Add to dashboard Cite

Hospital comparisons to improve quality of care require valid and reliable quality indicators. We aimed to test the validity and reliability of 6 breast cancer indicators by quantifying the influence of case-mix and random variation. Methods:The nationwide population-based database included 79 690 patients with breast cancer from 91 Dutch hospitals between 2011 and 2016. The indicator-scores calculated were: (1) irradical breast-conserving surgery (BCS) for invasive disease, (2) irradical BCS for ductal carcinoma-in-situ, (3) breast contour-preserving treatment, (4) magnetic resonance imaging (MRI) before neo-adjuvant chemotherapy, (5) radiotherapy for locally advanced disease, and (6) surgery within 5 weeks from diagnosis. Case-mix and random variation adjustments were performed by multivariable fixed and random effect logistic regression models. Rankability quantified the between-hospital variation, representing unexplained differences that might be the result of the level of quality of care, as low (,50%), moderate (50%-75%), or high (.75%).Results: All of the indicators showed between-hospital variation with wide (interquartile) ranges. Case-mix adjustment reduced variation in indicators 1 and 3 to 5. Random variation adjustment (further) reduced the variation for all indicators. Case-mix and random variation adjustments influenced the indicator-scores of individual hospitals and their ranking. Rankability was poor for indicator 1, 2, and 5, and moderate for 3, 4, and 6. Conclusions:The 6 indicators lacked validity and/or reliability to a certain extent. Although measuring quality indicators may stimulate quality improvement in general, comparisons and judgments of individual hospital performance should be made with caution if based on indicators that have not been tested or adjusted for validity and reliability, especially in benchmarking.

show abstract

Section: Discussionmentioning

confidence: 92%

Effect of Case-Mix and Random Variation on Breast Cancer Care Quality Indicators and Their Rankability

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…The same applies for low figures: our lowest ICC1, 0.068 corresponded with the lowest rankability, 0.144 (for polypharmacy). This relationship has been explored with a similar correlation for the ranking of binary hospital quality indicators [ 20 ]. Although an ICC1 of 0.05 has been regarded as the lower threshold for quality indicators, considering the rankability found in our study, this threshold might be higher, at roughly 0.15.…”

Section: Discussionmentioning

confidence: 99%

“…We also explored quality indicators’ rankability, i.e., the part of variability between nursing homes measured by quality indicators that results from true differences in quality of care [ 35 ]. High rankability for a particular indicator allows performance ranking for that indicator, e.g., polymedication [ 20 ]. Rankability (ρ) is defined as: ρ = VG /( VG + median ( s 2 )), with median ( s 2 ) indicating the variance of the individual facility effect estimates from a fixed effect regression model.…”

Section: Methodsmentioning

confidence: 99%

“…To assess each indicator’s capacity to differentiate between facilities, we used the ICC1, which reflects “the proportion of variance that is accounted for by the group level” [ 19 ]. While this method has been used in other settings to evaluate quality indicators’ between-provider variability (e.g., hospital quality indicators), it has not yet been used in long-term care contexts [ 20 ]. A second metric to assess between-provider variability is rankability, i.e., “the part of heterogeneity between … clinics [or nursing homes] that is due to true differences” [ 21 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing Nursing Homes Quality Indicators’ between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability

Favez

Zúñiga

Sharma

et al. 2020

IJERPH

View full text Add to dashboard Cite

Nursing home quality indicators are often used to publicly report the quality of nursing home care. In Switzerland, six national nursing home quality indicators covering four clinical domains (polypharmacy, pain, use of physical restraints and weight loss) were recently developed. To allow for meaningful comparisons, these indicators must reliably show differences in quality of care levels between nursing homes. This study’s objectives were to assess nursing home quality indicators’ between-provider variability and reliability using intraclass correlations and rankability. This approach has not yet been used in long-term care contexts but presents methodological advantages. This cross-sectional multicenter study uses data of 11,412 residents from a convenience sample of 152 Swiss nursing homes. After calculating intraclass correlation 1 (ICC1) and rankability, we describe between-provider variability for each quality indicator using empirical Bayes estimate-based caterpillar plots. To assess reliability, we used intraclass correlation 2 (ICC2). Overall, ICC1 values were high, ranging from 0.068 (95% confidence interval (CI) 0.047–0.086) for polypharmacy to 0.396 (95% CI 0.297–0.474) for physical restraints, with quality indicator caterpillar plots showing sufficient between-provider variability. However, testing for rankability produced mixed results, with low figures for two indicators (0.144 for polypharmacy; 0.471 for self-reported pain) and moderate to high figures for the four others (from 0.692 for observed pain to 0.976 for physical restraints). High ICC2 figures, ranging from 0.896 (95% CI 0.852–0.917) (self-reported pain) to 0.990 (95% CI 0.985–0.993) (physical restraints), indicated good reliability for all six quality indicators. Intraclass correlations and rankability can be used to assess nursing home quality indicators’ between-provider variability and reliability. The six selected quality indicators reliably distinguish care differences between nursing homes and can be recommended for use, although the variability of two—polypharmacy and self-reported pain—is substantially chance-driven, limiting their utility.

show abstract

“…Outcome measures are increasingly used to rank institutions inevitably giving the appearance of ranking quality of care [4]. Rankability measures the proportion of the variation between providers with regards to an outcome that is due to true differences; it is considered high if above 70% [34]. No study in this review directly assessed the rankability of LOS as an outcome measure in CAP.…”

Section: Discussionmentioning

confidence: 99%

Variation in clinical outcomes and process of care measures in community acquired pneumonia: a systematic review

2020

View full text Add to dashboard Cite

Background Variation in outcomes of patients with community acquired pneumonia (CAP) has been reported in some, but not all, studies. Although some variation is expected, unwarranted variation in healthcare impacts patient outcomes and equity of care. The aim of this systematic review was to: i) summarise current evidence on regional and inter-hospital variation in the clinical outcomes and process of care measures of patients hospitalised with CAP and ii) assess the strength of this evidence. Methods Databases were systematically searched from inception to February 2018 for relevant studies and data independently extracted by two investigators in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. Included studies enrolled adults hospitalised with CAP and reported a measure of variation between two or more units in healthcare outcomes or process of care measures. Outcomes of interest were mortality, length of hospital stay (LOS) and re-admission rates. A structured synthesis of the studies was performed. Results Twenty-two studies were included in the analysis. The median number of units compared across studies was five (IQR 4–15). Evidence for variation in mortality between units was inconsistent; of eleven studies that performed statistical significance testing, five found significant variation. For LOS, of nine relevant studies, all found statistically significant variation. Four studies reported site of admission accounted for 1–24% of the total observed variation in LOS. A shorter LOS was not associated with increased mortality or readmission rates. For readmission, evidence was mixed; of seven studies, 4 found statistically significant variation. There was consistent evidence for variation in the use of intensive care, obtaining blood cultures on admission, receiving antibiotics within 8 h of admission and duration of intravenous antibiotics. Across all outcome measures, only one study accounted for natural variation between units in their analysis. Conclusion There is consistent evidence of moderate quality for significant variation in length of stay and process of care measures but not for in-patient mortality or hospital re-admission. Evidence linking variation in outcomes with variation in process of care measures was limited; where present no difference in mortality was detected despite POC variation. Adjustment for natural variation within studies was lacking; the proportion of observed variation due to chance is not quantified by existing evidence.

show abstract

Ranking hospital performance based on individual indicators: can we increase reliability by creating composite indicators?

Cited by 18 publications

References 20 publications

Effect of Case-Mix and Random Variation on Breast Cancer Care Quality Indicators and Their Rankability

Effect of Case-Mix and Random Variation on Breast Cancer Care Quality Indicators and Their Rankability

Assessing Nursing Homes Quality Indicators’ between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability

Variation in clinical outcomes and process of care measures in community acquired pneumonia: a systematic review

Contact Info

Product

Resources

About