Are Simple Gain Scores Obsolete?

Abstract

It is widely believed that measures of gain, growth, or change, expressed as simple differences between pretest and posttest scores, are inherently unreliable. It is also believed that gain scores lack predictive validity with respect to other criteria. However, these conclusions are based on misleading assumptions about the values of parameters in familiar equations in classical test theory. The present paper examines modified equations for the validity and reliability of difference scores that describe applied testing situations more realistically and reveal that simple gain scores can be more useful in research than commonly believed.Over a quarter century ago, Cronbach & Furby (1970) and many other authors (e.g., Gulliksen, 1950;Lord & Novick, 1968) concluded that simple differences between pretest and posttest scores have questionable value in behavioral and social science research. Yet this conclusion seems incompatible with the intuition of researchers in many disciplines who assume that measures of gains, changes, differences, growth, and the like are meaningful in experimentation, program evaluation, educational accountability studies, and the investigation of developmental growth and change.During the past two decades, many researchers using difference (or gain) scores have had difficulty justifying the use of such measures, even when they appear to yield interesting and reproducible findings. However, recent research on this topic has provided results favorable to simple gain scores (e.

Determinates of youth and parent satisfaction in usual care psychotherapy

“…Prior to conducting the analyses examining potential determinants of satisfaction, change scores were computed to represent change in clinical outcomes from baseline to six-month follow-up by subtracting the six-month follow-up score from the baseline score for each measure. Although there is controversy regarding the use of change scores (also known as simple gain scores), there is support for use of this method for this purpose (Williams & Zimmerman, 1996;Zimmerman & Williams, 1998) and they have been used in well established evaluations of outcomes in children's mental health services .…”

Reciprocal relationship between proactive personality and work characteristics: A latent change score approach.

“…It is not directly measured and thus modeled as a latent variable with measurement errors of Y estimated. Albeit a full discussion of this approach of modeling change is beyond the scope of the present study, it should be noted that, although this method was once criticized, for example, by Lord (1956) as unreliable, later work has shown that the criticisms are not necessarily valid (Rogosa & Willett, 1983;Williams & Zimmerman, 1996). Indeed, recent work in psychology (McArdle, 2009), biology (Fitzmaurice, 2001), economics (Wooldridge, 2002), sociology (Halaby, 2004), and education (Willett, 1988) has recognized the appreciable reliability, validity, and usefulness of the change score approach.…”

Mental chronometry and individual differences: Modeling reliabilities and correlations of reaction time means and effect sizes

“…In classical test theory, the additional assumptions σ X = σ Y and ρ XX 0 ¼ ρ YY 0 can sometimes be made, although there are cases in which these assumptions are clearly inappropriate (Williams & Zimmerman, 1996). Within RT research, it seems reasonable that both of these assumptions would be true to a good approximation, because differences are usually computed between mean RTs obtained using very similar conditions.…”