A Comparative Study of IRT Fixed Parameter Calibration Methods

Abstract

This article provides technical descriptions of five fixed parameter calibration (FPC) methods, which were based on marginal maximum likelihood estimation via the EM algorithm, and evaluates them through simulation. The five FPC methods described are distinguished from each other by how many times they update the prior ability distribution and by how many EM cycles they use. Specifically, the five FPC methods included no prior weights updating and one EM cycle (NWU‐OEM) or multiple EM cycles (NWU‐MEM), one prior weights updating and one EM cycle (OWU‐OEM) or multiple EM cycles (OWU‐MEM), and multiple weights updating and multiple EM cycles (MWU‐MEM) methods. All the five FPC methods were evaluated in terms of recovery of the underlying ability distribution and item parameters. An important factor in the simulation was three different ability (normal) distributions—N(0, 1), N(0.5, 1.22), and N(1, 1.42)—for FPC groups, with the fixed item parameters obtained with a reference N(0, 1) group. Only the MWU‐MEM method appeared to perform properly under all the three distributions. Under the N(0, 1) distribution, the NWU‐MEM and OWU‐MEM methods also appeared to perform properly. Under the N(0.5, 1.22), and N(1, 1.42) distributions, however, the four methods other than the MWU‐MEM method resulted in some or severe under‐estimation in the recovery.

The Examination of the Classification of Students into Performance Categories by Two Different Equating Methods

“…Further research has supported the refinement proposed by Kim (2006) that could be implemented in PARSCALE. Several studies have investigated the ability of this method to capture the differences in the ability distributions of the nonequivalent groups and found that the refinement did correct for the bias in many circumstances (e.g., Baldwin et al, 2007;.…”

“…It was necessary to consider the performance of the FCIP methods when used over multiple years and under more realistic contexts. To differentiate the methods of implementing FCIP, FCIP-1 will refer to the typical way that FCIP is performed, and FCIP-2 will refer to the adjustment suggested by Kim (2006), which, in his study, led to improvements in the estimation of proficiency estimation when growth was present from one test administration to the next.…”

“…Because of the consistent result that FCIP underestimated differences in the ability distributions of examinees, Paek and Young (2005) and Kim (2006) both investigated methods to correct for this bias. Paek and Young introduced a concept of iterative prior updating, which resulted in a reduction of the bias inherent in the FCIP procedure.…”

Computerized Adaptive Test based on Item Response Theory in E-Learning System

“…Item response theory (IRT) is a psychometric model which is based on the idea that the probability of a correct response to an item is a mathematical function of person and item parameters (Kim, 2006). IRT provides a basis for estimating the parameters, determine how well the data conformed to the model and investigate how to measure the psychometric properties.…”

Educational Data Mining

“…The sample size recommended in the literature varies between 50 and 1000 persons (e.g. Kim 2006, Linacre 1994, Tsutakawa & Johnson 1990. Because IRT has been a prevalent CAT approach for decades, it seems logical to apply IRT for adaptive item sequencing in learning environments that consist of simple items.…”