When (Not) to Rely on the Reliable Change Index: A Critical Appraisal and Alternatives to Consider in Clinical Psychology

Abstract

The reliable change index (RCI) is a widely used statistical tool designed to account for measurement error when evaluating difference scores. Because of its conceptual simplicity and computational ease, it persists in research and applied psychology. However, researchers have repeatedly demonstrated ways that the RCI is insufficient or invalid for various applications. This is a problem in research and clinical psychology since this common tool is potentially problematic. The aims of this manuscript are to non-technically describe the formulation and assumptions of the RCI, to offer guidance as to when the RCI is (and is not) appropriate, and to identify what is needed for proper calculation of the RCI when it is used. Several criteria are identified to help determine whether the RCI is appropriate for a specific use. It is apparent that the RCI is the best available method only in a small number of situations, is frequently miscalculated, and produces incorrect inferences more often than simple alternatives, largely because it is highly insensitive to real changes. Specific alternatives are offered which may better operationalize common inferential tasks, including when more than two observations are available and when false negatives are equally costly to false positives.

Reliable change and the reliable change index: still useful after all these years?

“…the scales are assumed not to exhibit practice effects. In contrast, tests of learning and of cognitive performance are known to be affected by practice effects, so that an individual's scores will change over 2 Restricting the use of RC/RCI is consistent with the central argument of McAleavey (2021). At the time of writing this paper is available as an online pre-print, not yet formally published following peer review.…”

“…Over the years the JT method has been rigorously challenged, but it has survived, and can claim the virtue of simplicity and the recommendation of popularity relative to alternatives (Wise, 2004) and has received authoritative endorsement and recommendation for routine use (Lambert and Bailey, 2012; but see McAleavey, 2021 for reservations). Clearly, though, those using the JT method need to be aware of some of its weaknesses.…”

“…Given that norm data have been found, there is not much debate about selecting s 1 – it is the SD of an appropriate norm group on the scale (but see McAleavey, 2021) – but the selection of the reliability coefficient has been more contentious. Classical test theory offers several reliability coefficients, of which internal consistency and test–retest reliability ( r xx ) are the most relevant here.…”

“…Recently, in a comprehensive critique of the classical JT method, McAleavey (2021) has argued that only test–retest reliability estimated over short inter-test intervals should be used, because it is the measure of reliability matching the pre–post repeated measures aspect of the data being analysed, and coefficient α should not be used. If implemented, this recommendation would preclude the use of RC/RCI for scales for which appropriate test–retest reliability could not be located, likely to lead to considerable restriction in the use of RC/RCI.…”

“…Others concur, often emphasizing that the JT method is relatively simple to use and produces results that are often similar to those from more complex methods (e.g. Atkins et al ., 2005; Bauer et al ., 2004; Ronk et al ., 2016; but for reservations, see McAleavey, 2021).…”