Comparison of structural equation modelling, item response theory and Rasch measurement theory-based methods for response shift detection at item level: A simulation study

Abstract

When assessing change in patient-reported outcomes, the meaning in patients’ self-evaluations of the target construct is likely to change over time. Therefore, methods evaluating longitudinal measurement non-invariance or response shift at item-level were proposed, based on structural equation modelling or on item response theory. Methods coming from Rasch measurement theory could also be valuable. The lack of evaluation of these approaches prevents determining the best strategy to adopt. A simulation study was performed to compare and evaluate the performance of structural equation modelling, item response theory and Rasch measurement theory approaches for item-level response shift detection. Performances of these three methods in different situations were evaluated with the rate of false detection of response shift (when response shift was not simulated) and the rate of correct response shift detection (when response shift was simulated). The Rasch measurement theory-based method performs better than the structural equation modelling and item response theory-based methods when recalibration was simulated. Consequently, the Rasch measurement theory-based approach should be preferred for studies investigating only recalibration response shift at item-level. For structural equation modelling and item response theory, the low rates of reprioritization detection raise issues on the potential different meaning and interpretation of reprioritization at item-level.

Performance of a Rasch-based method for group comparisons of longitudinal change and response shift at the item level in PRO data: A simulation study

“…These good performances may be related to the Bonferroni correction applied in the two iterative steps (step C and step 3) following likelihood ratio tests and to the iterative steps themselves. This asset of ROSALI has already been shown in a previous simulation study [14]. In fact, if LRT wrongly concludes to the presence of recalibration or difference in item difficulties between groups at time 1, often no items will be identified in the following iterative step, hence, the wrong decision of LRT is corrected in the final model of ROSALI.…”

“…HRQoL). They were compared recently [14] and the method based on Rasch Measurement Theory (RMT) models, called the RespOnse Shift ALgorithm at Item level (ROSALI) showed better performances compared to other methods for detecting and accounting for recalibration RS in the measurement of PRO change. However, in this version of ROSALI it is assumed that the majority of patients experiences RS the same way.…”

“…Performances of methods for RS detection have rarely been assessed and conducting simulation studies have been recommended to fill this gap [11,12]. Although the performances of ROSALI without a covariate were satisfactory in a previous simulation study [14], the effects of the major changes made to extend ROSALI for longitudinal studies with two groups need to be assessed.…”

Response shift in patient-reported outcomes: definition, theory, and a revised model

“…As these latent variable models allow to formally specify and estimate the measurement model between the target construct (as latent variable(s)) and the measure (e.g. the items) using a set of equations, a test verifying whether this set of equations can be assumed equivalent at each time of measurement can be seen as a formal test of the violation of the PCI [45][46][47]. Sébille et al's critical review of the literature also demonstrated that there are other response shift methods that also examine discrepancies between target change and observed change [12].…”

Response shift in results of patient-reported outcome measures: a commentary to The Response Shift—in Sync Working Group initiative

“…The remaining methods rely on various statistical methods. Within the framework of latent variable models, methods include Structural Equation Modeling (SEM) [13], Item Response Theory [33] and Rasch Measurement Theory [34]. Other frameworks not necessarily requiring modeling of latent variables encompass Relative Importance Analysis [35], Classification and Regression Tree [36], Random Forest Regression [37], and Mixed Models and Growth Mixture Models [38].…”