Multidimensional nonadditivity in one-facet g-theory designs: A profile analytic approach.

Abstract

We introduce a new method for estimating the degree of nonadditivity in a one-facet generalizability theory design. One-facet G-theory designs have only one observation per cell, such as persons answering items in a test, and assume that there is no interaction between facets. When there is interaction, the model becomes nonadditive, and G-theory variance estimates and reliability coefficients are likely biased. We introduce a multidimensional method for detecting interaction and nonadditivity in G-theory that has less bias and smaller error variance than methods that use the one-degree of freedom method based on Tukey's test for nonadditivity. The method we propose is more flexible and detects a greater variety of interactions than the formulation based on Tukey's test. Further, the proposed method is descriptive and illustrates the nature of the facet interaction using profile analysis, giving insight into potential interaction like rater biases, DIF, threats to test security, and other possible sources of systematic construct-irrelevant variance. We demonstrate the accuracy of our method using a simulation study and illustrate its descriptive profile features with a real data analysis of neurocognitive test scores.

Change Score and Subscore Precision and Reliability of the Children’s Depression Inventory

“…The first coefficient, ρ ^ D , was the classical approach, based on the G-theory single facet design (which is equivalent to Cronbach’s alpha, a measure of internal consistency). The second was the updated reliability coefficient ρ ^ Λ ( profile reliability) for ranking CDI change scores based on Grochowalski et al’s (2022) method for detecting additional interaction variance, thus increasing change score reliability. However, as change scores are typically not used for ranking, but rather as absolute measures of change, we also computed the G-theory change score dependability estimate Φ ^ Λ ( profile dependability ), which provides information about change score precision for making decisions about individual change scores.…”

“…Considering the warnings from psychometricians about the typically low reliability of change scores, it is important to note that the change score analysis used here deviates from the classical methods to address the concerns of change scores. Classical methods of change score tend to underestimate reliability because they assume no item-by-person interactions, but such interactions are common and add interaction variance to the scores that goes ignored by the classical methods (Grochowalski et al, 2022). In other words, methods like Cronbach's alpha (Cronbach, 1951) consider interaction variance to be error, thus decreasing reliability.…”

“…The second estimate we calculated was the relative change score reliability-referred to here as the profile reliability, which does include systematic variance due to interactions and correlated errors (Grochowalski et al, 2022;Gu et al, 2018). The estimate is based on randomeffects ANOVA factorization of scores-referred to as Generalizability Theory (Brennan, 2001;Cronbach et al, 1972)-such that items and persons are each random factors.…”

“…However, interaction variance is not error, so classical methods like Cronbach’s alpha tend to underestimate reliability and return pessimistically low change score reliability estimates (Gu et al, 2018). An accurate method of change score reliability estimation needs to account for these interactions, which we argue are common patterns of responses that can influence scores systematically (Grochowalski et al, 2022; Kim & Grochowalski, 2019). Because of this, we estimate change score reliability using reliability coefficients that account for interactions, we estimate the classical coefficient for comparison, and we detect and illustrate the profile of responses that—if ignored—would result in the classical underestimate of change score reliability.…”

Detecting Group Collaboration Using Multiple Correspondence Analysis

“…Unusual group activity appears as a "bump" anomaly from the funnel, as depicted by the filled circles in Figure 1b. We use the context of group cheating to describe the emergence of the bump, but such an abnormality as depicted in Figure 1b emerges when systematic variance due to local item dependence (Yen 1984(Yen , 1993 and model nonadditivity (Grochowalski et al, 2022) exists. Figure 1a depicts local independence, as the horizontal dimension should only contain systematic variability due to person ability, while the vertical dimension should be nothing more than random residuals.…”