Score Reporting for Examinees with Incomplete Data on Large‐Scale Educational Assessments

Abstract

Technical difficulties occasionally lead to missing item scores and hence to incomplete data on computerized tests. It is not straightforward to report scores to the examinees whose data are incomplete due to technical difficulties. Such reporting essentially involves imputation of missing scores. In this paper, a simulation study based on data from three educational tests is used to compare the performances of six approaches for imputation of missing scores. One of the approaches, based on data mining, is the first application of its kind to the problem of imputation of missing data. The approach based on data mining and a multiple imputation approach based on chained equations led to the most accurate imputation of missing scores, and hence to most accurate score reporting. A simple approach based on linear regression performed the next best overall. Several recommendations are made regarding the reporting of scores to examinees with incomplete data.

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

“…Importantly, the missingness process is summarized by the latent response variable. As an alternative, multiple imputation at the level of items can be employed to handle missing item responses properly [ 22 , 23 ]. However, scoring missing item responses as wrong could be defended for validity reasons [ 24 , 25 , 26 ].…”

Disrupted Data: Using Longitudinal Assessment Systems to Monitor Test Score Quality

“…In this article, we show how these recent features of test data motivate the need for a longitudinal approach to examining data quality. Though multiple approaches exist to address missing data within a tested population (e.g., Peugh & Enders, 2004) and within individual item response vectors at a single test occasion (e.g., Sinharay, 2021), a key inference for test scores, especially in a pandemic, is about educational progress or decline for populations over time. The Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014) stress the importance of describing the target population and collecting information on testing conditions for assessing validity evidence.…”

Estimating Probabilities of Passing for Examinees With Incomplete Data in Mastery Tests

“…Huisman and Molenaar (2001) compared several missing-data imputation approaches for imputation of the total/raw score in psychology tests in the presence of missing item scores. Sinharay (2021) compared several missing-data imputation approaches for imputation of scaled scores in the presence of missing item scores in educational tests. Smits et al (2002) compared several missing-data imputation approaches with respect to their accuracy in imputation of grade point averages in the presence of missing grades on several courses.…”

“…Measurement professionals are familiar with the phenomenon of missing item scores, or, incomplete tests, on educational assessments. A wide variety of problems related to missing item scores in educational or psychological measurement were tackled by researchers such as De Ayala et al (2001), Finch (2008), Feinberg (2020), Sinharay (2021), Smits et al (2002), and Xiao and Bulut (2020).…”