The Effects of Correlated Errors on Generalizability and Dependability Coefficients

Abstract

This study investigated the effects of correlated errors on the person x occasion design in which the confounding effect of equal time intervals results in correlated error terms in the linear model. Two specific error correlation structures were examined: the first-order stationary autoregressive (SARI), and the first-order nonstationary autoregressive (NARI) with increasing variance parameters. The effects of correlated errors on the existing generalizability and dependability coefficients were assessed by simulating data with known variances (six different combinations of person, occasion, and error variances), occasion sizes, person sizes, correlation parameters, and increasing variance parameters. Estimates derived from the simulated data were compared to their true values. The traditional estimates were acceptable when the error terms were not correlated and the error variances were equal. The coefficients were underestimated when the errors were uncorrelated with increasing error variances. However, when the errors were correlated with equal vanances the traditional formulas overestimated both coefficients. When the errors were correlated with increasing variances, the traditional formulas both overestimated and underestimated the coefficients. Finally, increasing the number of occasions sampled resulted in more improved generalizability coefficient estimates than dependability coefficient estimates. Index terms: changing error variances, computer simulation, correlated errors, dependability coefficients, generalizability coefficients. In the past 20 years, generalizability theory has emerged as an important method of analyzing the reliability or generalizability of test scores or observational data when multiple sources of variation occur simultaneously. Traditionally, the reliability of test scores was measured by classical test theory statistics such as the Pearson product-moment correlation coefficient and coefficient alpha, which often could not adequately account for all sources of variation. Ebel (1951), Horst (1949), Hoyt (1941), and Medley & Mitzel (1958) showed that classical test theory estimates of reliability could be written in terms of the ratio of mean squares derived using analysis of variance (ANOVA) techniques. Cronbach, Gleser, Nanda, & Rajaratnam (1972) demonstrated that by using ANOVA mean squares for models with several effects, reliability estimates could be derived for multiple sources of variation (e.g., occasions, items, and raters) in a testing situation. The study of such reliability estimates is known as generalizability theory. Random or mixed effects ANOVA methods are used to estimate the variance components in generalizability theory. The ANOVA mean squares are derived for each effect in the model and set equal to their expectations. The variance components then are determined by algebraic manipulation of these equations. Finally, lizability coefficients (p~), which reflect the ratios of various combinations of these variance components, are estimated. Consequently, some of...

Coping With Memory Effect and Serial Correlation When Estimating Reliability in a Longitudinal Framework

“…Our conclusions fully coincide with the results found by Smith and Luecht (1992) and Bost (1995) in their study about the effect of ignoring a stationary correlated error structure on the estimation of the G-coefficients. This misspecification can seriously affect both, the point estimates of the reliability parameters and the inferential procedures related to T R and Λ R .…”

“…These findings fully coincide with the results reported by Smith and Luecht (1992) and Bost (1995) in their studies of the effect of ignoring a stationary correlated error structure on the estimation of the G-coefficients. Fortunately, unlike in the modeling framework used in G-theory, linear mixed models allow for the absorption of such a correlation structure.…”

“…In their simulations, these authors considered a stationary correlated error structure, i.e., the error terms were allowed correlated but still with equal variances over time. Bost (1995) studied this issue further by examining the effect of both stationary and non-stationary auto-regressive error variance-covariance matrices. His results showed that, in the presence of non-stationary auto-regressive error, the G-coefficients were usually underestimated and the magnitude of the bias increased with the number of observations.…”

An Examination of G-Theory Methods for Modeling Multitrait–Multimethod Data

“…For example, the univariate G-theory literature has suggested that unaccounted for covariances among residual effects would lead to inflated estimates of person main effect variance and to underestimates of residual variance (e.g., Bost, 1995;Smith & Luecht, 1992). The effects predicted by the work of these researchers, coupled with Box's (1954) earlier work, would also suggest that unaccounted for covariance among other sources of observed variance (e.g., Person  Trait and Person  Method interaction effects) would lead to inflated estimates of person main effects in the context of applying a univariate G-theory model to MTMM data because it would be the person main effect term that is relied upon to account for such covariance.…”

Responsible Modeling of Measurement Data for Appropriate Inferences: Important Advances in Reliability and Validity Theory

“…Similar results have been found for g-theory designs with multiple time points. In such designs, underestimation was present for uncorrelated errors with increasing variances over time, overestimation was present for correlated errors with equal variances over time, and both directions of estimation bias were present for correlated errors with unequal variances over time (Bost, 1995). It is important to note that correlated errors may arise for a variety of reasons.…”