Adapting the multilevel model for estimation of the reliable change index (RCI) with multiple timepoints and multiple sources of error

Abstract

Objective One of the primary tools in the assessment of individual‐level patient outcomes is Jacobson and Truax, (1991’s) Reliable Change Index (RCI). Recent efforts to optimize the RCI have revolved around three issues: (a) extending the RCI beyond two timepoints, (b) estimating the RCI using scale scores from item response theory or factor analysis and (c) estimation of person‐ and time‐specific standard errors of measurement. Method We present an adaptation of a two‐stage procedure, a measurement error‐corrected multilevel model, as a tool for RCI estimation (with accompanying Statistical Analysis System syntax). Using DASS‐21 data from a community‐based mental health center (N = 379), we illustrate the potential for the model as unifying framework for simultaneously addressing all three limitations in modeling individual‐level RCI estimates. Results Compared to the optimal‐fitting RCI model (moderated nonlinear factor analysis scoring with measurement error correction), an RCI model that uses DASS‐21 total scores produced errors in RCI inferences in 50.8% of patients; this was largely driven by overestimation of the proportion of patients with statistically significant improvement. Conclusion Estimation of the RCI can now be enhanced by the use of latent variables, person‐ and time‐specific measurement errors, and multiple timepoints.

Determining the reliable feature change in longitudinal radiomics studies: A methodological approach using the reliable change index

“…The proposed RCI could also be extended for multi-time-point delta-radiomics. 45 In data illustration, the reliability of longitudinal feature changes was only demonstrated with respect to image acquisition.…”

“…Regarding the RCI model, 23, 24 there are many variants of the RCI model 44 to be further investigated. The proposed RCI could also be extended for multi‐time‐point delta‐radiomics 45 . In data illustration, the reliability of longitudinal feature changes was only demonstrated with respect to image acquisition.…”

Reliable change and the reliable change index: still useful after all these years?

“…Others concur, often emphasizing that the JT method is relatively simple to use and produces results that are often similar to those from more complex methods (e.g. Atkins et al, 2005;Bauer et al, 2004;Ronk et al, 2016;but (Morgan-Lopez et al, 2022). As yet, these methods are too new to have received much evaluative attention, but for investigators with large data samples and the requisite technical expertise, these may become useful techniques.…”

“…IRT is an alternative to classical test theory for psychometric test development. Other examples have explored ways to extend the calculation of RC/RCI across multiple time points, using moderated, non-linear factor analysis to estimate SE M (Morgan-Lopez et al ., 2022). As yet, these methods are too new to have received much evaluative attention, but for investigators with large data samples and the requisite technical expertise, these may become useful techniques.…”

Differential symptom weighting in estimating empirical thresholds for underlying PTSD severity: Toward a “platinum” standard for diagnosis?

“…However, it has been long understood that total scores can be problematic from a psychometric perspective (McNeish & Wolf, 2020) as well as a clinical perspective (Saavedra et al., 2022; Saavedra, Morgan‐López, Hien, Back et al., 2021). Total scoring is primarily thought of as a “data management problem”, often with little‐to‐no consideration for what is implied psychometrically by its use (Morgan‐López, Hien et al., 2022; Morgan‐López, Saavedra et al., 2022). For any latent variable model that is structured under the “equal weights” model (e.g., 1‐parameter logistic item response theory (IRT), nonlinear factor analysis [NLFA] with equal factor loadings), the total score is a sufficient statistic for maximum likelihood estimation of the latent variable (Andrich, 1978, 2018; McNeish & Wolf, 2020).…”