Unlike ROC analysis, a new IRT method identified clinical thresholds unbiased by disease prevalence

Terluin, Berend; Griffiths, Philip; Wouden, Johannes C. van der; Ingelsrud, Lina Holm; Terwee, Caroline B.

doi:10.1016/j.jclinepi.2020.05.008

Cited by 17 publications

(22 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The PASS values we derived with other methods were 2-5 points higher than those derived using the primary analysis method. The overestimation of PASS values calculated with the ROC method is associated with the proportions of patients with satisfactory symptom levels exceeding 50% (Terluin et al 2020). Advantages of using the adjusted predictive modeling method include that we are able to overcome the issue of biased PASS values in the direction of the largest group (Terluin et al 2017).…”

Section: Discussionmentioning

confidence: 99%

Which Oxford Knee Score level represents a satisfactory symptom state after undergoing a total knee replacement?

et al. 2020

Self Cite

View full text Add to dashboard Cite

Background and purpose — Meaningful interpretation of postoperative Oxford Knee Score (OKS) levels is challenging. We established Patient Acceptable Symptoms State (PASS) and Treatment Failure (TF) values for the OKS in patients undergoing primary total knee replacement (TKR) in Denmark. Patients and methods — Data from patients undergoing primary TKR between February 2015 and January 2019 was extracted from the arthroplasty registry at the Copenhagen University Hospital, Hvidovre in Denmark. Data included 3, 12, and 24 months postoperative responses to the OKS and 2 anchor questions asking whether they considered their symptom state to be satisfactory, and if not, whether they considered the treatment to have failed. PASS and TF threshold values were calculated using the adjusted predictive modeling method. Non-parametric bootstrapping was used to derive 95% confidence intervals (CI). Results — Complete 3, 12, and 24 months postoperative data was obtained for 187 of 209 (89%), 884 of 915 (97%), and 575 of 586 (98%) patients, with median ages from 68 to 70 years (59 to 64% female). 72%, 77%, and 79% considered as having satisfactory symptoms, while 6%, 11%, and 11% considered the treatment to have failed, at 3, 12, and 24 months postoperatively, respectively. OKS PASS values (CI) were 27 (26–28), 30 (29–31), and 30 (29–31) at 3, 12, and 24 months postoperatively. TF values were 27 (26–28) and 27 (26–29) at 12 and 24 months postoperatively. Interpretation — The OKS PASS values can be used to guide the interpretation of TKR outcome and support quality assessment in institutional and national registries.

show abstract

Section: Discussionmentioning

confidence: 99%

Which Oxford Knee Score level represents a satisfactory symptom state after undergoing a total knee replacement?

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Smaller samples and larger variability in the COA change score reduces the reliability of the results produced here using ROC analyses, regardless of anchor correlation. Although not assessed in this study, Terluin et al (2020) described how the MCT derived from ROC curve analyses may be biased when anchor groups are unequal in size [15]. However, here we show that PM methods (as described elsewhere) [14] were better able to accurately represent the true MCT at the individual-level, even with weaker anchor correlations.…”

Section: Discussionmentioning

confidence: 51%

How strong should my anchor be for estimating group and individual level meaningful change? A simulation study assessing anchor correlation strength and the impact of sample size, distribution of change scores and methodology on establishing a true meaningful change threshold

et al. 2022

Self Cite

View full text Add to dashboard Cite

Purpose: Treatment bene t as assessed using clinical outcome assessments (COAs), is a key endpoint in many clinical trials at both the individual and group level. Anchor-based methods can aid interpretation of COA change scores beyond statistical signi cance, and help derive a meaningful change threshold (MCT). However, evidence-based guidance on the selection of appropriately related anchors is lacking. Methods: A simulation was conducted which varied sample size, change score variability and anchor correlation strength to assess the impact of these variables on recovering the true simulated MCT at both the individual and group-level. At the individual-level, Receiver Operating Characteristic (ROC) curves and Predictive Modelling (PM) anchor analyses were conducted. At the group-level, group means of the 'notimproved' and 'improved' groups were compared.Results: Sample sizes, change score variability and magnitude of anchor correlation affected accuracy of the estimated MCT. At the individual-level, ROC curves were less accurate than PM methods at recovering the true MCT. For both methods, smaller samples led to higher variability in the returned MCT, but higher variability still using ROC. Anchors with weaker correlations with COA change scores had increased variability in the estimated MCT. An anchor correlation of 0.50-0.60 identi ed a true MCT cut-point under certain conditions using ROC. However, anchor correlations as low as 0.30 were appropriate when using PM under certain conditions. At the group-level, the MCT was consistently underestimated regardless of the anchor correlation.Conclusion: Findings show that the chosen method, sample size and variability in change scores in uence the necessary anchor correlation strength when identifying a true individual-level MCT. Often, this needs to be higher than the commonly accepted threshold of 0.30. Stronger correlations than 0.30 are required at the group-level, but a speci c recommendation is not provided. Results can be used to assist researchers selecting and assessing the quality of anchors.

show abstract

“…Whereas the new method identified MWIC values between 3 and 4 across the three follow-up points, the traditional MWIC estimates showed much more variability, and bias related to the proportion of improved patients. That the ROC-based MWIC is biased by the proportion improved is well known [14,26]. The adjusted MWIC is supposed to adjust for this bias [14], but the present findings suggest that the adjusted MWIC is also biased by the proportion improved.…”

Section: Discussionmentioning

confidence: 54%

Establishing Thresholds for Meaningful Within-individual Change Using Longitudinal Item Response Theory

Bjørner

Terluin

Trigg

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

PURPOSE: Thresholds for meaningful within-individual change (MWIC) are useful for interpreting patient-reported outcome measures (PROM). Transition ratings (TR) have been recommended as anchors to establish MWIC. Traditional statistical methods for analyzing MWIC such as mean change analysis, receiver operating characteristic (ROC) analysis, and predictive modeling ignore problems of floor/ceiling effects and measurement error in the PROM scores and the TR item. We present a novel approach to MWIC estimation for multi-item scales using longitudinal item response theory (LIRT).METHODS: A Graded Response LIRT model for baseline and follow-up PROM data was expanded to include a TR item measuring latent change. The LIRT threshold parameter for the TR established the MWIC threshold on the latent metric, from which the observed PROM score MWIC threshold was estimated. We compared the LIRT approach and traditional methods using an example data set with baseline and three follow-up assessments differing by magnitude of score improvement, variance of score improvement, and baseline-follow-up score correlation.RESULTS: The LIRT model provided good fit to the data. LIRT estimates of observed PROM MWIC varied between 3 and 4 points score improvement. In contrast, results from traditional methods varied from 2 points to 10 points - strongly associated with proportion of self-rated improvement. Best agreement between methods was seen when approximately 50% rated their health as improved.CONCLUSION : Results from traditional analyses of anchor-based MWIC are impacted by study conditions. LIRT constitutes a promising and more robust analytic approach to identifying thresholds for MWIC.

show abstract

Unlike ROC analysis, a new IRT method identified clinical thresholds unbiased by disease prevalence

Cited by 17 publications

References 20 publications

Which Oxford Knee Score level represents a satisfactory symptom state after undergoing a total knee replacement?

Which Oxford Knee Score level represents a satisfactory symptom state after undergoing a total knee replacement?

How strong should my anchor be for estimating group and individual level meaningful change? A simulation study assessing anchor correlation strength and the impact of sample size, distribution of change scores and methodology on establishing a true meaningful change threshold

Establishing Thresholds for Meaningful Within-individual Change Using Longitudinal Item Response Theory

Contact Info

Product

Resources

About