Examination of Different Item Response Theory Models on Tests Composed of Testlets

Abstract

The purpose of this research is to first estimate the item and ability parameters and the standard error values related to those parameters obtained from Unidimensional Item Response Theory (UIRT), bifactor (BIF) and Testlet Response Theory models (TRT) in the tests including testlets, when the number of testlets, number of independent items, and sample size change, and then to compare the obtained results. Mathematic test in PISA 2012 was employed as the data collection tool, and 36 items were used to constitute six different data sets containing different numbers of testlets and independent items. Subsequently, from these constituted data sets, three different sample sizes of 250, 500 and 1000 persons were selected randomly. When the findings of the research were examined, it was determined that, generally the lowest mean error values were those obtained from UIRT, and TRT yielded a mean of error estimation lower than that of BIF. It was found that, under all conditions, models which take into consideration the local dependency have provided a better model-data compatibility than UIRT, generally there is no meaningful difference between BIF and TRT, and both models can be used for those data sets. It can be said that when there is a meaningful difference between those two models, generally BIF yields a better result. In addition, it has been determined that, in each sample size and data set, item and ability parameters and correlations of errors of the parameters are generally high.

Comparison of Testlet Effect on Parameter Estimates Using Different Item Response Theory Models

“…DeMars (2006), in his research with PISA 2000 data, used both mathematics and reading literacy data to examine the ability estimations of the independent item model and testlet effect model and stated that the correlations between these estimations were close to 1. A similar result has been observed in other studies (Baghaei & Ravand, 2016;Eckes, 2014;Eckes & Baghaei, 2015;Yılmaz Kogar & Kelecioglu, 2017).…”

“…The current study was determined that the results obtained from the standard IRT model and the TRT model are quite close to each other. This result is similar to the studies conducted on the real data set (Baghaei & Ravand, 2016;Demars, 2006;Eckes, 2014;Eckes & Baghaei, 2015;Özdemir, 2017;Yılmaz Kogar & Kelecioglu, 2017). The reason why the result is this way is probably the small variance of the testlet in the data set used in this study because Glas et al (2000) stated that the testlet effect variances lower than .50 had a negligible effect on the results.…”

“…It has been thought that the use of standard IRT models for these tests may be insufficient since the LID assumption has been violated in the tests involving testlets. Therefore TRT models have become a frequently used model in research to testlet effect (DeMars, 2006;Eckes, 2014;Geramipour, 2021;Min & He, 2014;Özdemir, 2017;Paap & Veldkamp, 2012;Wainer & Wang, 2000;Yılmaz Kogar & Kelecioglu, 2017). Glas, Wainer, and Bradlow (2000) examined in their simulation study that when the testlet effect was ignored and the standard IRT model was used, the mean absolute errors of discrimination and difficulty parameter estimation were poorly predicted.…”

The Assumption Testing and Compatibility Model of Final Examination in Islamic Religious Education by Item Response Theory

“…IRT is said to have two basic assumptions which are unidimensionality and local item independence [5], and in order to determine the appropriate IRT model, first it should be assessed whether the data set at hand meets the IRT assumptions or not [6]. This assumption states that the conditional probability of observing a response pattern given a particular latent trait value equals the product of the items' conditional probabilities [7].…”