Examination of Testlet Effect in Open-Ended Items

Abstract

The testlets are prevalently used in national and international standard tests and the teacher-made achievement tests thanks to their practicality. The purpose of the current study is to evaluate the testlet effect in test consisting the open-ended items and to identify the possible errors when this effect is ignored. The data consists of the scores given by three raters of the responses of 380 students in the teacher training program to two open-ended items in two testlets. The two designs with and without the testlet effect were modeled (totally crossed random design and the nested random design) and the differences between them were analyzed by Generalizability Theory (GT). The results showed that raters had a slight effect on the scores in both models tested, and the reliability of the scores was overestimated when the testlet effect was not ignored in such measurement tools.

Investigating The Effect Of Testlets Consisting Of Open-Ended And Multiple-Choice Items On Reliability Via Generalizability Theory

“…Some rules, such as the comprehensibility of the material, the ability to respond to the items correctly based only on the material, and careful determination of the number of items, should be considered in developing testlets (Tekin, 2009). Although there is no definite rule in determining the number of items that should be included in testlets (Kaya Uyanık & Ertuna, 2022), there may be between 2-12 items (Yaman, 2016) depending on the characteristics of the common material. In determining the number of items in testlets, the characteristics of the structure shared by the items and content validity can also be taken into consideration.…”

“…In the literature, there are studies in which testlets are handled with Item Response Theory (Sireci et al, 1991;Wainer & Thissen, 1996) and G theory (Lee & Frisbie, 1999;Lee et al, 2000;Taşdelen Teker, 2014;Kaya Uyanık & Gelbal, 2018;Kaya Uyanık & Ertuna, 2022). In addition, while Gessaroli and Folske (2002) addressed testlets with factor analysis, Kaya Uyanık and Gelbal (2018) studied two-facet patterns with the Generalizability approach in Item Response modeling using testlet data generated in simulation in their study and compared the results obtained with the results from G theory.…”