A Test Can Have Multiple Reliabilities

Abstract

It is argued that the generalizability theory interpretation of coefficient alpha is important. In this interpretation, alpha is a slightly biased but consistent estimate for the coefficient of generalizability in a subjects x items design where both subjects and items are randomly sampled. This interpretation is based on the “domain sampling” true scores. It is argued that these true scores have a more solid empirical basis than the true scores of Lord and Novick (1968), which are based on “stochastic subjects” (Holland, 1990), while only a single observation is available for each within-subject distribution. Therefore, the generalizability interpretation of coefficient alpha is to be preferred, unless the true scores can be defined by a latent variable model that has undisputed empirical validity for the test and that is sufficiently restrictive to entail a consistent estimate of the reliability—as, for example, McDonald’s omega. If this model implies that the items are essentially tau-equivalent, both the generalizability and the reliability interpretation of alpha can be defensible.

Is It Really More Robust? Comparing the Robustness of the Structural After Measurement (SAM) Approach to Structural Equation Modeling (SEM) Against Local Model Misspecifications with Alternative Estimation Approaches

“…However, we think that Cronbach's alpha or its stratified variants offer conceptual advantages because they do not rely on a factor model. In contrast, they are only based on an exchangeability assumption of items (see also [43]), and items can be regarded as random instead of fixed ( [44]; but see also [45]). Unfortunately, it is frequently noted that the computation of Cronbach's alpha would require that a one-dimensional factor model must fit the data [46] without referring to the origins of the exchangeability concept behind Cronbach's alpha.…”

Some thoughts on analytical choices in the scaling model for test scores in international large-scale assessment studies

“…The test items should cover the ability domain defined by the test framework (test blueprint; see also Pellegrino & Chudowsky, 2003;Reckase, 2017). It might be legitimate to assume that there exists a larger population of test items (henceforth, labeled by I ) from which the items are chosen in a particular study, and true ability values would be defined as outcomes in a study in which all items from the population would have been chosen (Cronbach & Shavelson, 2004; see also Ellis, 2021, Kane, 1982Brennan, 2001). Interestingly, it has been argued that classical test theory (CTT) or generalizability theory (GT; Cronbach et al, 1963) treats items in a study as random and, as a consequence, allows the inference to a larger set of items in a population of items (see also Nunnally & Bernstein, 1994;Markus & Borsboom, 2013).…”

“…such a design-based perspective, no assessment of the model fit for the set of item responses x n is required. For example, the use of Cronbach's alpha (Cronbach, 1951) as a reliability measure for the sum score does not require that a model with equal item loadings and uncorrelated residual errors have to fit the data of item responses (Cronbach, 1951;Cronbach & Shavelson, 2004;Ellis, 2021;Meyer, 2010;Nunnally & Bernstein, 1994;Tryon, 1957). In the same manner, as for persons, resampling methods for items can be used to determine standard errors in estimated abilities (Liou & Yu, 1991;Wainer & Thissen, 1987;Wainer & Wright, 1980) by resampling items or groups of items for which abilities are reestimated (see also Michaelides & Haertel, 2014).…”

Reconceptualization of Coefficient Alpha Reliability for Test Summed and Scaled Scores

“…Coefficient alpha has specific definitions of both true scores and error scores in the reliability context. Ellis (2021) reported that coefficient alpha can be differently interpreted using the definition of true scores in three test theories: classical test theory, generalizability theory, and latent trait theory.…”