Generalizability Theory: An Analysis of Variance Approach to Measurement Problems in Educational Assessment

Abstract

Of increasing interest to the educational assessment researchers is the reliability of scores obtained by various measurement procedures like tests, rating scales, surveys, and observation forms. Traditional methods of reliability based on Classical Test Theory (CTT) consider only one source of measurement errors. Generalizability Theory (GT) extends CTT by providing a flexible and practical framework for estimating the effects of multiple sources of measurement errors through an application of analysis of variance procedures. It helps educational assessment researchers to determine appropriate conditions in terms of items, occasions, raters conducive to obtaining an optimal level of score reliability. This paper highlights the utility and applicability of the GT analysis in the educational assessment research. The paper begins with a brief history of the GT tracing its growth and development and then discusses its defining features and advantages and disadvantages. The paper illustrates the application of the GT in the educational assessment research. The paper closes with a recommendation for educational assessment researchers to take advantage of the GT in the development of measurement procedures and further research agenda in this area.

A comparison of the reliability of the Solo- and revised Bloom's Taxonomy-based classifications in the analysis of the cognitive levels of assessment questions

“…The aim of G studies is to predict the variance components related to the measurement population (Brennan, 2001). In other words, G studies are carried out in order to obtain information about the source of variance influencing the generalizability of the observations (Alkharusi, 2012). The second stage of G theory analyses is D studies.…”

“…G çalışmalarının amacı, ölçme evreni ile ilgili olan varyans bileşenleri hakkında tahminler elde etmektir (Brennan, 2001). Bir başka deyişle G çalışmaları, gözlemlerin genellenebilirliğini etkileyen değişkenlik kaynaklarına ilişkin bilgi edinmek amacıyla gerçekleştirilir (Alkharusi, 2012). G kuramı analizlerinin ikinci adımını K çalışmaları oluşturmaktadır.…”

The Impact of Item Parceling Ratios and Strategies on the Internal Structure of Assessment Center Ratings

“…Second, it can inform AC practitioners about the size of these sources of variance, which can then be applied to future AC design. Third, it allows researchers to have flexibility in how to treat variance, and therefore, it is possible to determine if, and in what circumstances, variance can be generalized (Alkharusi, 2012; Jackson et al, 2016; Shavelson & Webb, 1991). Therefore, G theory may indeed offer a more robust procedure for dealing with misspecification and model termination issues that tend to plague CFA.…”

How well aligned are common core textbooks to students’ development in area measurement?

“…Each LT, CCSSM, and textbook code was placed in one cell using topics in Table to compute the alignment index. For reliable results, 20% of the items were coded by the authors using Polikoff’s () technique, a generalizability theory D study (Alkhrausi, ), which produced reliable results of 0.964. Based on the initial coding agreement, the authors coded all of the textbook items jointly to produce a final set of tables for analysis.…”