The Optimization of Generalizability Studies with Resource Constraints

Abstract

Generalizability theory is a measurement theory that provides a framework for examining the dependability of behavioral measurements. When limited resources are available determining the appropriate number of conditions to use in a measurement design is not a simple task. This paper presents a methodology for determining the optimal number of observations to use in a measurement design when resource constraints are imposed.

A Practical Solution to Optimizing the Reliability of Teaching Observation Measures Under Budget Constraints

“…To minimize relative error subject to the cost constraint, cn l n r B, we find the stationary points of the LaGrange function, F(n l , n r , l) = s 2 (d) À l(cn l n r B): Marcoulides and Goldstein (1990) showed these points to be…”

“…An alternative is to numerically optimize the LaGrange function through computer intensive methods or adjust the expression for relative error variance in some way to simplify the problem. Marcoulides and Goldstein (1990) chose the latter and obtained an upper bound to the relative error variance by eliminating facet sample size terms from the equation. Their choice of terms to eliminate was arbitrary and can lead to optimized relative error variance that notably differs from the numerical solution.…”

“…Although their work produced equations for optimal sample sizes in two-and three-facet crossed designs, it does not easily extend to more than three facets or nested designs (Saunders, Theunissen, & Baas, 1989). More feasible solutions involve discrete optimization (Saunders et al, 1989), the Cauchy-Schwartz inequality (Saunders, 1992), or the LaGrange multiplier method (Goldstein & Marcoulides, 1991;Marcoulides & Goldstein, 1990).…”

Evaluating Prototype Tasks and Alternative Rating Schemes for a New ESL Writing Test through G-theory

“…When costs for conditions for various measurement facets (e.g., development cost per task, cost per rater) included in a given measurement design are known in advance, an optimization algorithm can be used to compute maximum reliability coefficients that take into account budget constraints for the testing process(Marcoulides & Goldstein, 1990;1992). However, since the current study was a prototyping study in which the main focus of the study were to evaluate a range of possible task types and several rating schemes, the costs for tasks, test administration, and raters for operational tests were not yet fully known at the time the study was undertaken.…”

Generalizability Theory: An Analysis of Variance Approach to Measurement Problems in Educational Assessment

“…Performing a generalizability analysis to pinpoint the sources of measurement error allows the researcher to determine how many conditions of each facet are needed (e.g., number of items, number of occasions) to obtain an optimal level of generalizability (Marcoulides & Goldstein, 1990). The items, in the design presented in Table 2, represent a source of measurement error.…”