Using Structural Equation Modeling to Reproduce and Extend ANOVA-Based Generalizability Theory Analyses for Psychological Assessments

Abstract

Generalizability theory provides a comprehensive framework for determining how multiple sources of measurement error affect scores from psychological assessments and using that information to improve those assessments. Although generalizability theory designs have traditionally been analyzed using analyses of variance (ANOVA) procedures, the same analyses can be replicated and extended using structural equation models. We collected multi-occasion data from inventories measuring numerous dimensions of personality, self-concept, and socially desirable responding to compare variance components, generalizability coefficients, dependability coefficients, and proportions of universe score and measurement error variance using structural equation modeling versus ANOVA techniques. We further applied structural equation modeling techniques to continuous latent response variable metrics and derived Monte Carlo-based confidence intervals for those indices on both observed score and continuous latent response variable metrics. Results for observed scores estimated using structural equation modeling and ANOVA procedures seldom varied. Differences in reliability between raw score and continuous latent response variable metrics were much greater for scales with dichotomous responses, thereby highlighting the value of doing analyses on both metrics to evaluate gains that might be achieved by increasing response options. We provide detailed guidelines for applying the demonstrated techniques using structural equation modeling and ANOVA-based statistical software.

Applying multivariate generalizability theory to psychological assessments.

“…Historically, GT has been used predominantly with subjectively scored performance assessments but is equally applicable to objectively scored self-report and other measures. Applications of univariate GT with self-reports have been comprehensively addressed in recent studies for scales considered individually or as part of a profile and underscore the importance of taking multiple sources of measurement error into account when measuring psychological traits (Vispoel et al, 2018a, 2018b, 2018c, 2018d, 2019; Vispoel, Hong, et al, in press; Vispoel, Lee, Chen, & Hong, 2023a; Vispoel & Tao, 2013; Vispoel, Xu, & Kilinc, 2021; Vispoel, Xu, & Schneider, 2022a, 2022b). Our goal here was to demonstrate how multivariate GT can replicate such analyses for individual subscales, while simultaneously extending them to composite scores derived from those subscales.…”

“…Factor loadings for both models are set equal to 1, and variance and covariance components involving relative error are directly estimated. Jorgensen (2021) (also see Vispoel, Hong, et al, in press; Vispoel, Hong, & Lee, 2023; Vispoel, Lee, Chen, & Hong, 2023a) demonstrated that variance components for absolute differences in scores can be obtained indirectly by imposing effect coding and other constraints within each subscale. When using effect coding (see Little et al, 2006), factor loadings are constrained to average 1, and the sum of item intercepts to equal zero.…”

“…In addition to handling missing data directly, GT-based structural equation models allow for conducting univariate and multivariate GT analyses on continuous latent response variable metrics that correct for scale coarseness effects resulting from limited numbers of response options and/or unequal intervals between those options using robust diagonally weighted least squares, paired maximum likelihood, or other appropriate estimators (see, e.g., Ark, 2015; Jorgensen, 2021; Vispoel et al, 2019; Vispoel, Lee, & Hong, in press; Vispoel, Hong, & Lee, 2023; Vispoel, Lee, Chen, & Hong, 2023a). 8 Together, such corrections for scale coarseness and missing data within structural equation modeling packages represent clear advantages over mGENOVA , which permits neither.…”

The psychometric upgrade psychophysiology needs

“…Gtheory equations are built around estimates of variance, but how variance is estimated is left up to the researcher. Some approaches use traditional ANOVA procedures (Brennan, 2001(Brennan, , 2003, structural equation models (SEMs; Vispoel et al, 2023;Vispoel et al, 2018a), or multilevel models (Brennan, 2010;Li, 2023). Variance components can be estimated with these procedures, and then these components can be used in equations for psychometric reliability estimates (see Table 1 for leading to the development of coefficient alpha and his thoughts on alpha 50 years after the publication of the seminal work.…”

Editorial for the Special Issue “Computational Aspects and Software in Psychometrics II”

“…The article by Vispoel, Lee, Chen, and Hong [7], titled "Using structural equation modeling to reproduce and extend ANOVA-based generalizability theory analyses for psychological assessments", discusses the estimation of statistical models in generalizability theory (GT) using structural equation models (SEM). It compares SEM-based estimation with estimation based on analysis of variance (ANOVA) models.…”