The choice between different statistical approaches to risk-adjustment influenced the identification of outliers

Fedeli, Ugo; Brocco, Stefano; Alba, Natalia; Rosato, Rosalba; Spolaore, Paolo

doi:10.1016/j.jclinepi.2006.11.017

Cited by 8 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…empirical Bayesian methods, hierarchical/multilevel models, and regression trees [ 4 – 6 , 8 , 12 ]. Comparisons of methods with respect to estimation or hospital outlier detection have been done for selected medical conditions and by simulations [ 26 – 28 ]. Multilevel methods have been reported to be more conservative than methods based on fixed effects [ 5 , 11 ][ 9 ] and to have convergence problems [ 9 ].…”

Section: Discussionmentioning

confidence: 99%

Observed to expected or logistic regression to identify hospitals with high or low 30-day mortality?

et al. 2018

View full text Add to dashboard Cite

IntroductionA common quality indicator for monitoring and comparing hospitals is based on death within 30 days of admission. An important use is to determine whether a hospital has higher or lower mortality than other hospitals. Thus, the ability to identify such outliers correctly is essential. Two approaches for detection are: 1) calculating the ratio of observed to expected number of deaths (OE) per hospital and 2) including all hospitals in a logistic regression (LR) comparing each hospital to a form of average over all hospitals. The aim of this study was to compare OE and LR with respect to correctly identifying 30-day mortality outliers. Modifications of the methods, i.e., variance corrected approach of OE (OE-Faris), bias corrected LR (LR-Firth), and trimmed mean variants of LR and LR-Firth were also studied.Materials and methodsTo study the properties of OE and LR and their variants, we performed a simulation study by generating patient data from hospitals with known outlier status (low mortality, high mortality, non-outlier). Data from simulated scenarios with varying number of hospitals, hospital volume, and mortality outlier status, were analysed by the different methods and compared by level of significance (ability to falsely claim an outlier) and power (ability to reveal an outlier). Moreover, administrative data for patients with acute myocardial infarction (AMI), stroke, and hip fracture from Norwegian hospitals for 2012–2014 were analysed.ResultsNone of the methods achieved the nominal (test) level of significance for both low and high mortality outliers. For low mortality outliers, the levels of significance were increased four- to fivefold for OE and OE-Faris. For high mortality outliers, OE and OE-Faris, LR 25% trimmed and LR-Firth 10% and 25% trimmed maintained approximately the nominal level. The methods agreed with respect to outlier status for 94.1% of the AMI hospitals, 98.0% of the stroke, and 97.8% of the hip fracture hospitals.ConclusionWe recommend, on the balance, LR-Firth 10% or 25% trimmed for detection of both low and high mortality outliers.

show abstract

Section: Discussionmentioning

confidence: 99%

Observed to expected or logistic regression to identify hospitals with high or low 30-day mortality?

et al. 2018

View full text Add to dashboard Cite

show abstract

“…25 There are multiple nuances of hierarchical models that have also been shown to affect the number of outliers, almost always reducing the number of outliers in comparison to non-multilevel approaches. 32,33 The various criteria employed with logistic regression models resulted in a range of outliers for general surgery morbidity (7 to 36), general surgery mortality (0 to 10), colorectal morbidity (4 to 10), and colorectal mortality (0 to 1). The overall number of outliers differed between the general surgery and colorectal groups primarily due to the smaller sample sizes per hospital (median number of cases per hospital: 951 vs. 149), where smaller samples lead to wider confidence intervals and fewer definitive outliers, as well as different event rates and differences in intra-institutional correlation.…”

Section: Discussionmentioning

confidence: 99%

Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs

Bilimoria

Cohen

Merkow

et al. 2010

Journal of Gastrointestinal Surgery

View full text Add to dashboard Cite

There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.

show abstract

“…Fixed effects logistic regression ( L R F ) assumes that all variation between providers is due to differences in case-mix and that the model specification is correct. By including providers as dummy variables, direct comparisons between providers can be made [ 34 , 35 ]. Random effects logistic regression ( L R R ) accounts for the increased similarity between patients attending the same provider, the hierarchical structure of the data, and allows for residual variance between providers that may not be attributable to performance.…”

Section: Methodsmentioning

confidence: 99%

“…As provider effects are assumed to come from an underlying distribution in L R R , effect estimates of providers (especially those with low volume) can borrow information from the other providers, shrinking these effects towards the mean of all providers [ 34 ]. This results in the identification of fewer performance outliers as compared to when L R F is used [ 35 – 40 ]. Given the fundamental difference in how the model is formulated, the decision whether to use L R F or L R R is largely dependent on the goal of the profiling exercise.…”

Section: Methodsmentioning

confidence: 99%

Outlier classification performance of risk adjustment methods when profiling multiple providers

Brakenhoff

Roes

Moons

et al. 2018

BMC Med Res Methodol

View full text Add to dashboard Cite

BackgroundWhen profiling multiple health care providers, adjustment for case-mix is essential to accurately classify the quality of providers. Unfortunately, misclassification of provider performance is not uncommon and can have grave implications. Propensity score (PS) methods have been proposed as viable alternatives to conventional multivariable regression. The objective was to assess the outlier classification performance of risk adjustment methods when profiling multiple providers.MethodsIn a simulation study based on empirical data, the classification performance of logistic regression (fixed and random effects), PS adjustment, and three PS weighting methods was evaluated when varying parameters such as the number of providers, the average incidence of the outcome, and the percentage of outliers. Traditional classification accuracy measures were considered, including sensitivity and specificity.ResultsFixed effects logistic regression consistently had the highest sensitivity and negative predictive value, yet a low specificity and positive predictive value. Of the random effects methods, PS adjustment and random effects logistic regression performed equally well or better than all the remaining PS methods for all classification accuracy measures across the studied scenarios.ConclusionsOf the evaluated PS methods, only PS adjustment can be considered a viable alternative to random effects logistic regression when profiling multiple providers in different scenarios.

show abstract

The choice between different statistical approaches to risk-adjustment influenced the identification of outliers

Cited by 8 publications

References 23 publications

Observed to expected or logistic regression to identify hospitals with high or low 30-day mortality?

Observed to expected or logistic regression to identify hospitals with high or low 30-day mortality?

Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs

Outlier classification performance of risk adjustment methods when profiling multiple providers

Contact Info

Product

Resources

About