Detecting Nonadditivity in Single‐Facet Generalizability Theory Applications: Tukey's Test

Abstract

Under the generalizability‐theory (G‐theory) framework, the estimation precision of variance components (VCs) is of significant importance in that they serve as the foundation of estimating reliability. Zhang and Lin advanced the discussion of nonadditivity in data from a theoretical perspective and showed the adverse effects of nonadditivity on the estimation precision of VCs in 2016. Contributing to this line of research, the current article directs the discussion of nonadditivity from a theoretical perspective to a practical application and highlights the importance of detecting nonadditivity in G‐theory applications. To this end, Tukey's test for nonadditivity is the only method to date that is appropriate for the typical single‐facet G‐theory design, in which a single observation is made per element within a facet. The current article evaluates the Type I and Type II error rates of Tukey's test. Results show that Tukey's test is satisfactory in controlling for falsely detecting nonadditivity when the data are actually additive and that it is generally powerful in detecting nonadditivity when it exists. Finally, the article demonstrates an application of Tukey's test in detecting nonadditivity in a judgmental study of educational standards and shows how Tukey's test results can be used to correct imprecision in the estimated VC in the presence of nonadditivity.

Development of the Breast Cancer Related Lymphedema Self-Care Scale

“…Tukey's test was utilized to evaluate scores for the scale and its subscales. The results of this test was not statistically significant, which means that the scale has the additivity feature (Lin & Zhang, 2018). The Hoteling T 2 test was used and the Response Bias of the scale was found to be T 2 = 70.40, p = .00.…”

The use of generalizability theory in investigating the score dependability of classroom-based L2 reading assessment

“…As such, although the variance components for the interaction effect observed in this study were evident, it is unclear how much random error contributed to the total score variance. This confounding nature, which is also discussed in Lin and Zhang’s (2018) study, is a limitation of current G theory designs. Second, the majority of the participants in the study were French and Arabic speakers whose reading proficiency levels ranged from low- to high-intermediate.…”

Multidimensional nonadditivity in one-facet g-theory designs: A profile analytic approach.

“…They further found that person true score variance estimates were often underestimated when additive models were assumed. Lin and Zhang (2018) conducted simulation analyses to determine the Type-I and Type-II error rates for the one-degree of freedom test when applied to G-theory one-facet designs. They found that their test was generally well powered when detecting nonadditivity, and that it had an acceptable Type-I error rate when the data was additive.…”