A Method for Analyzing Sparse Data Matrices in the Generalizability Theory Framework

Abstract

In generalizability analyses, unstable, and potentially invalid, variance component estimates may result from using only a limited portion of available data. However, missing observations are common in operational performance assessment settings because of the nature of the assessment design. This article describes a procedure for overcoming the computational and technological limitations in analyzing data with missing observations by extracting data from a sparsely filled data matrix into analyzable smaller subsets of data. This subdividing method is accomplished by creating data sets that exhibit structural designs that are common in generalizability analyses, namely, the crossed, MBIB, and nested designs. The validity of this subdividing method is examined using a Monte Carlo simulation. The method is demonstrated on an operational data set. Index terms: analysis of variance (ANOVA), crossednested-MBIB design, generalizability theory and method, interrater reliability, large-scale analysis, missing data, performance assessment, rater design, variance component.In recent years, performance assessment has become popular as a means for assessing students because these assessments provide direct measures of nontraditional student outcomes. Generalizability theory (G theory), developed by Cronbach, Gleser, and Rajaratnam (1963), is often used in the development of performance assessments to identify the relative strengths of multiple sources of measurement error and to make projections concerning how to increase score reliability. Because these assessments are time-consuming to administer and score, examinees seldom respond to all test items and raters seldom evaluate all examinee responses. As a result, a common problem encountered by those using G theory with large-scale performance assessments is working with sparse data matrices (i.e., missing data). The purpose of this article is to develop a method for analyzing data sets with missing observations. The authors examined this method in relation to rater inconsistencies, number of examinees tested, number of raters employed, and degree of standardization in distributing tasks to raters. These factors are discussed in detail in the subsequent sections.The authors first described the technical problems caused by missing observations in performance assessment data sets. Then they reviewed some common approaches used to overcome these missing data problems and the limitations of these approaches. Next, they discussed a new G theory technique, followed by an illustration of how to restructure and analyze a hypothetical sparsely filled data set so that it can be accommodated by currently available analytic methods. A Monte Carlo simulation was used to evaluate the statistical properties of this new method. Finally, the authors applied their methods to a data set coming from a large-scale direct writing assessment and presented the results of these analyses in terms of the comparability of the methods.

Investigating the Utility of Analytic Scoring for the Toefl Academic Speaking Test (Tast)

“…Thus, very limited information could be obtained regarding errors associated with raters. So that all effects could be estimated, an alternative analysis was conducted to treat each rater pair as a block, estimate it as a fully crossed design, and then pool variances across the rater pairs (for references on pooling variance components, see Chiu, 2000;Chiu & Wolfe, 2002;Smith, 1980;Wiley, 1992). This analysis allows us to use all the data in both Phase 1 and Phase 2 and to examine the impact of all sources of error on score dependability independently.…”

“…These averaged variance and covariance components can then be used in D studies to yield more accurate estimates of score dependability given the alternative measurement designs specified. This method is especially preferable for scenarios where multiple rating schemes are utilized, as in the present study, in which different rating designs (different G study structures) were used in Phase 1 and Phase 2 (Chiu & Wolfe, 2002). Pooling variance and covariance components from all G studies would allow us to keep all the data in the analysis.…”

“…Pooling variance and covariance components from all G studies would allow us to keep all the data in the analysis. Otherwise, tossing out valuable data could result in unstable and inaccurate variance and covariance component estimates (Chiu & Wolfe, 2002). …”

Designing and Scaling Level-Specific Writing Tasks in Alignment With the CEFR: A Test-Centered Approach

“…As we see next, this difference is also captured in the results of the multifaceted Rasch modeling. Similar to Chiu and Wolfe (2002), the aggregated effects combine substantial variation across booklets highlighting the importance of comparing all available data instead of using just one particular booklet, for example.…”

“…We then aggregated the resulting variance components across the 13 booklets as described in Chiu and Wolfe (2002). The proportion of ratings in each booklet was used as the weight, which varied across booklets as a different number of tasks were included in each booklet due to different difficulty levels of the tasks in each booklet.…”