Observed score and true score equating procedures for multidimensional item response theory

Abstract

The purpose of this research was to develop observed score and true score equating procedures to be used in conjunction with the Multidimensional Item Response Theory (MIRT) framework. Currently, MIRT scale linking procedures exist to place item parameter estimates and ability estimates on the same scale after separate calibrations are conducted. These procedures account for indeterminacies in (1) translation, (2) dilation, (3) rotation, and (4) correlation. However, no procedures currently exist to equate number correct scores after parameter estimates are placed on the same scale. This research sought to fill this void in the current psychometric literature. Three equating procedures-two observed score procedures and one true score procedure-were created and described in detail. One observed score procedure was presented as a direct extension of unidimensional IRT observed score equating, and is referred to as the "Full MIRT Observed Score Equating Procedure." The true score procedure and the second observed score procedure incorporated the statistical definition of the "direction of best measurement" in an attempt to equate exams using unidimensional IRT (UIRT) equating principles. These procedures are referred to as the "Unidimensional Approximation of MIRT True Score Equating Procedure" and the "Unidimensional Approximation of MIRT Observed Score Equating Procedure," respectively. Three exams within the Iowa Test of Educational Development (ITED) Form A and Form B batteries were used to conduct UIRT observed score and true score equating, MIRT observed score and true score equating, and equipercentile equating. The equipercentile equating procedure was conducted for the purpose of comparison since this procedure does not explicitly violate the IRT assumption of unidimensionality. Results indicated that the MIRT equating procedures performed more similarly to the equipercentile equating procedure than the UIRT equating procedures, presumably due to the violation of the unidimensionality assumption under the UIRT equating procedures. Future ACKNOWLEDGMENTS I would like to thank my dissertation advisors, Dr. Timothy Ansley and Dr. Won-Chan Lee, for their help and guidance throughout the dissertation writing process. Dr. Timothy Ansley always provided invaluable feedback pertaining to my dissertation work. Countless hours were spent in Dr. Won-Chan Lee's office, collaborating on research ideas, determining mathematical solutions, and being delivered coffee from fellow graduate students in the department. I would also like to thank my dissertation members, Dr. Michael Kolen, Dr. Robert Brennan, and Dr. Dale Zimmerman for their service in regards to my dissertation progress. Having been given the opportunity to view my dissertation research through their eyes has definitely opened up many ideas that otherwise would never have entered my mind. I would like to thank my parents, Fred and Nancy Brossman, for all of the support that they provided along the way. I certainly would not have completed my doctoral studies i...

Desafios e perspectivas da implementação computacional de testes adaptativos multidimensionais para avaliações educacionais

“…Podem ser atribuídos diferentes setups de teste para diferentes grupos de examinados. As quatro menções desenvolvidas não podem ser utilizadas em situações de avaliação somativa classificatória, pois estas requerem uma extensa análise da dimensionalidade do banco de itens, a construção e interpretação de escalas na MIRT e os processos de equalização (equating) e ligação (linking), detalhadas em Brossman (2010). A construção de escalas geralmente é realizada por equipes multidisciplinares em longos períodos de desenvolvimento.…”

Simple structure MIRT equating for multidimensional tests

“…This results in a particular true score on a new form corresponding to a nearly infinite number of combinations, indicating that there is no unique solution in establishing the true-score relationship between two forms. As an alternative approach, a true-score unidimensional approximation of the MIRT equating procedure was proposed by Brossman and Lee (2013). For constrained MIRT models such as the bi-factor model, it is possible to use just single general ability as the reference scale (Lee et al, 2015).…”

“…Interestingly, none of the previous studies on MIRT equating under the RG design implemented MIRT scale linking (Brossman & Lee, 2013;Lee & Brossman, 2012;Peterson, 2014;Tao & Cao, 2016). Some have avoided this issue by using the orthogonal multivariate normal ability distribution and the Identity variancecovariance matrix.…”

“…However, with the growing realization that a test is inherently multidimensional and requires multiple distinct abilities to reach a correct answer, the research is rapidly expanding to include various MIRT frameworks in equating. So far, the MIRT models presented in the equating literature include (a) simple-structure (SS-MIRT) models (Lee & Brossman, 2012), (b) full-MIRT models (Brossman & Lee, 2013;Lee, Lee, & Brennan, 2014;Peterson & Lee, 2014), (c) bi-factor (BF-MIRT) models (Kim, Lim, & Lee, in press;Lee, et al, 2016;Peterson & Lee, 2014), and (d) testlet response (TR-MIRT) models (Kim et al, in press;Tao & Cao, 2016). In this section, the first part mainly focuses on a review of SS-MIRT equating and the second part presents a summary of the literature related to other MIRT equating procedures.…”

“…In spite of an emerging scholarly perspective supporting the use of MIRT models, only a limited number of studies have been carried out in MIRT equating, as summarized in (Brossman & Lee, 2013), (b) true-score unidimensional approximation of MIRT equating (Brossman & Lee, 2013), and (c) observed-score unidimensional approximation of MIRT equating (Brossman & Lee, 2013), (d) bi-factor MIRT observed-score equating , (e) bi-factor MIRT true-score equating ), (f) testlet response model MIRT observed-score equating (Tao & Cao, 2016), and (g) testlet response model MIRT true-score equating (Tao & Cao, 2016). These methods utilize different MIRT models or different procedure types (i.e., observed-score vs. true-score), but take a similar position in that they all recognize the multidimensional nature of educational tests.…”