The Reliability Factor: Modeling Individual Reliability with Multiple Items from a Single Assessment

Abstract

Reliability is a crucial concept in psychometrics. Although it is typically estimated as a single fixed quantity, previous work suggests that reliability can vary across persons, groups, and covariates. We propose a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. The proposed method employs a “Reliability Factor” that models the error variance of each case across multiple indicators, thereby producing case-specific reliability estimates. Additionally, we use Gaussian process modeling to estimate a nonlinear, non-monotonic function between the latent factor itself and the reliability of the measure, providing an analogue to test information functions in item response theory. The reliability factor model is a new tool for examining latent regions with poor conditional reliability, and correlates thereof, in a classical test theory framework.

Measuring the Uncertainty of Imputed Scores

“…The dependence of the reliability-and SEM-like measures (that were suggested in this paper) on the examinees' missing data patterns may be concerning to some because reliability and SEM are often considered as population-level measures (e.g., Martin & Rast, 2022). However, one could view the reliability-like measures for an individual with missing scores on items, say, 36-40, to represent the reliability of a population of examinees whose scores on items 36-40 were missing and an imputation approach was used to impute their scores.…”

“…However, one could view the reliability-like measures for an individual with missing scores on items, say, 36-40, to represent the reliability of a population of examinees whose scores on items 36-40 were missing and an imputation approach was used to impute their scores. In addition, as Quails (1996), Feldt, Steffen, &Gupta (1985), Lek and Van De Schoot (2018), and Martin and Rast (2022) argued, a measure may be more reliable or less error-prone for some individuals than others, or, equivalently, either of reliability and SEM can vary across, individuals, groups, or covariates. So, the idea of the variation of the reliability-or SEM-like measures over examinees is not unprecedented in the measurement literature.…”