Item Position and Item Difficulty Change in an IRT-Based Common Item Equating Design

Abstract

In operational testing programs using item response theory (IRT), item parameter invariance is threatened when an item appears in a different location on the live test than it did when it was field tested. This study utilizes data from a large state's assessments to model change in Rasch item difficulty (RID) as a function of item position change, test level, test content, and item format. As a follow-up to the real data analysis, a simulation study was performed to assess the effect of item position change on equating. Results from this study indicate that item position change significantly affects change in RID. In addition, although the test construction procedures used in the investigated state seem to somewhat mitigate the impact of item position change, equating results might be impacted in testing programs where other test construction practices or equating methods are utilized.

Modeling Item‐Position Effects Within an IRT Framework

“…Traditionally, two kinds of item‐position effects have been discerned (Kingston & Dorans, ): a practice or a learning effect occurs when the items become easier in later positions, and a fatigue effect occurs when items become more difficult if placed towards the end of the test. Recent empirical studies on the effect of item position include Hohensinn et al (), Meyers, Miller, and Way (), Moses, Yang and Wilson (), Pommerich and Harris (), and Schweizer, Schreiner and Gold ().…”

Student, School, and Country Differences in Sustained Test-Taking Effort in the 2009 PISA Reading Assessment

“…In these studies, exarninee effort is cornrnonly seen as constant throughout the assessrnent. However, research has shown that perforrnance in large-scale assessrnents can decrease (e.g., Hohensinn et al, 2008;Meyers, Miller, & Way, 2009), possibly due to fatigue or a decline in rnotivation. Hence, it seerns likely that, during testing, a change in examinee effort can take place.…”

A structural equation modeling approach for examining position effects in large-scale assessments

“…For example, the difficulty of an item can increase in later positions due to a fatigue effect or decreasing test-taking effort (Hohensinn et al 2011;Weirich et al 2016). To investigate item position effects, researchers have proposed different approaches using logistic regression (e.g., Davey and Lee 2011;Pomplun and Ritchie 2004), multilevel IRT models based on the GLMM framework (e.g., Albano 2013; Li et al 2012;Weirich et al 2014), and test equating (e.g., Pommerich and Harris 2003; Meyers et al 2009;Store 2013). The purpose of the current study was to introduce a factor analytic approach for modeling item position effects using the SEM framework.…”

“…To date, three different methodological approaches for investigating item position effects have been described: (1) logistic regression models (e.g., Davey and Lee 2011;Pomplun and Ritchie 2004;Qian 2014); (2) multilevel models based on the generalized linear mixed modeling (GLMM) framework (e.g., Albano 2013; Alexandrowicz and Matschinger 2008;Debeer and Janssen 2013;Hartig and Buchholz 2012;Li et al 2012;Weirich et al 2014); and (3) test equating methods (e.g., Kingston and Dorans 1984;Kolen and Harris 1990;Moses et al 2007;Pommerich and Harris 2003;Meyers et al 2009;Store 2013). Although there are some empirical studies that used the factor analytic methods for modeling position effects (e.g., Bulut et al 2016;Schweizer 2012;Schweizer et al 2009), the current study represents the first study that utilized the SEM framework as a methodological approach for examining item position effects.…”

“…For example, in a reading assessment where items are often connected to a reading passage, it is challenging to maintain the original positions of the items from field test to final form to eliminate the possibility of any position effects (Meyers et al 2009). Similarly, the positions of items cannot be controlled in computerized adaptive tests in which item positions typically differ significantly from one examinee to another (Davey and Lee 2011;Kolen and Brennan 2014;Meyers et al 2009). Therefore, non-negligible item position effects may become a source of error in the estimation of item parameters and the latent trait Hahne 2008).…”