A Comparison of Different Psychometric Approaches to Modeling Testlet Structures: An Example with C‐Tests

Schroeders, Ulrich; Robitzsch, Alexander; Schipolowski, Stefan

doi:10.1111/jedm.12054

Cited by 16 publications

(18 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recent research addressed the suitability of psychometric approaches for the analysis and evaluation of C-tests with a focus on the issue of local dependence within texts (Eckes, 2006(Eckes, , 2011Eckes & Baghaei, 2015;Schroeders, Robitzsch, & Schipolowski, 2014). Local dependence violates a critical psychometric assumption, stating that an examinee's response to an item (gap) does not affect the probability of the examinee's response to another item (gap) within the same text.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Eckes¹

2016

Language Testing

View full text Add to dashboard Cite

This paper presents an approach to standard setting that combines the prototype group method (PGM; Eckes, 2012) with a receiver operating characteristic (ROC) analysis. The combined PGM-ROC approach is applied to setting cut scores on a placement test of English as a foreign language (EFL). To implement the PGM, experts first named learners whom they considered to be typical of each of five levels of language proficiency as specified by the Common European Framework of Reference for Languages (CEFR; Council of Europe, 2001). Out of a total of 3,310 examinees taking different trial versions of the placement test, 470 learner prototypes were identified. For this set of prototypes, Rasch model estimates of EFL proficiency served as input to a series of ROC analyses, one for each pair of adjacent proficiency levels. Cut scores were derived using the Youden index that maximizes the overall rate of correct classification and minimizes the overall rate of misclassification. Findings confirmed that this method allows for the setting of cut scores that show a high level of classification accuracy in terms of the correspondence with expert categorizations of examinee prototypes. In addition, the ROC-based cut scores were associated with higher classification accuracy than cut scores derived from a logistic regression analysis of the same data. Potential further uses and implications of the PGM-ROC approach in the context of language testing and assessment are discussed.

show abstract

Section: Discussionmentioning

confidence: 99%

“… 4. Copula models are marginal IRT models that represent local dependencies within testlets (texts) as correlated residuals (Braeken, 2011; Schroeders et al, 2014). …”

mentioning

confidence: 99%

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Eckes¹

2016

Language Testing

View full text Add to dashboard Cite

show abstract

“…A testlet is a set of items clustered around the same stimuli ( Wainer and Kiely, 1987 ; Rosenbaum, 1988 ). This effect results from violations of the local item independence (LII) assumption, a fundamental assumption related to classical test theory (CTT) and item response theory (IRT) ( Yen, 1993 ; Schroeders et al, 2014 ; Raykov et al, 2015 ). This assumption is a prerequisite for IRT unidimensional models ( de Ayala, 2009 , p. 20) and has been the focus in simulation (e.g., Bradlow et al, 1999 ) and applied studies for IRT (e.g., Schroeders et al, 2014 ; Baldonado et al, 2015 ).…”

Section: A Motivational-developmental Free Response Assessment Through a Bifactor Lensmentioning

confidence: 99%

A Motivational-Developmental Free Response Assessment Through a Bifactor Lens

Alpizar

French

2021

Front. Psychol.

View full text Add to dashboard Cite

The Motivational-Developmental Assessment (MDA) measures a university student’s motivational and developmental attributes by utilizing overlapping constructs measured across four writing prompts. The MDA’s format may lead to the violation of the local item independence (LII) assumption for unidimensional item response theory (IRT) scoring models, or the uncorrelated errors assumption for scoring models in classical test theory (CTT) due to the measurement of overlapping constructs within a prompt. This assumption violation is known as a testlet effect, which can be viewed as a method effect. The application of a unidimensional IRT or CTT model to score the MDA can result in imprecise parameter estimates when this effect is ignored. To control for this effect in the MDA responses, we first examined the presence of local dependence via a restricted bifactor model and Yen’s Q3 statistic. Second, we applied bifactor models to account for the testlet effect in the responses, as this effect is modeled as an additional latent variable in a factor model. Results support the presence of local dependence in two of the four MDA prompts, and the use of the restricted bifactor model to account for the testlet effect in the responses. Modeling the testlet effect through the restricted bifactor model supports a scoring inference in a validation argument framework. Implications are discussed.

show abstract

“…Zu beachten ist, dass die einzelnen Items in C-Tests nicht notwendig lokal stochastisch unabhängig sind (Harsch und Hartig 2010;Schroeders et al 2014). Allerdings scheinen diese Abhängigkeiten zumeist eher klein zu sein und sich in erster Linie auf Itemschwierigkeiten und Reliabilitäten auszuwirken (Schroeders et al 2014 Lediglich ein Item lud nicht signifikant auf den FP-Faktor (p = 0,326).…”

Section: Statistische Analysenunclassified

“…Allerdings scheinen diese Abhängigkeiten zumeist eher klein zu sein und sich in erster Linie auf Itemschwierigkeiten und Reliabilitäten auszuwirken (Schroeders et al 2014 Lediglich ein Item lud nicht signifikant auf den FP-Faktor (p = 0,326). Gleichzeitig zeigten sich nur noch für 6 der 40 Items signifikante Ladungen auf den UFP-Faktor, ein weiteres Item verpasste das Signifikanzniveau knapp (p = 0,064).…”

Section: Statistische Analysenunclassified

Analyse der Aufgaben zur Evaluation der Bildungsstandards in Physik - Differenzierung von schriftsprachlichen Fähigkeiten und Fachlichkeit

Härtig

Heitmann²,

Retelsdorf

2015

Z Erziehungswiss

View full text Add to dashboard Cite

Die Daten für die hier präsentierten Analysen stammen aus dem Projekt ManKobE ("Mathematisch-naturwissenschaftliche Kompetenzen in der beruflichen Erstausbildung"), das aus Mitteln der Leibniz-Gemeinschaft gefördert wird (Förderkennzeichen: SAW−2012-IPN−2). ZusammenfassungBei der Evaluation der Bildungsstandards in den naturwissenschaftlichen Fächern sieht das Itemdesign vor, benötigtes Vorwissen innerhalb der Testung vorzugeben und damit auch im Sinne einer Kompetenzorientierung Umgang mit Fachwissen zu evaluieren. Dies setzt voraus, dass die Schülerinnen und Schüler die Informationen angemessen rezipieren. Es wird daher angenommen, dass schriftsprachliche Fähigkeiten eine zentrale Rolle für die Erfassung physikbezogener Kompetenzen in Leistungstests spielen, zumal einige Items schriftliche Antworten verlangen. Der vorliegende Beitrag geht daher der Frage nach, in welchem Umfang schriftsprachliche Fähig-keiten im Zusammenhang mit den Ergebnissen der Evaluation der Bildungsstandards im Kompetenzbereich Umgang mit Fachwissen des Fachs Physik steht. Die Studie zeigt an einer Stichprobe von N = 1961 Auszubildenden mittels Vergleich logistischer Modelle auf, dass die Items der Bildungsstandards von schriftsprachlichen Fähigkeiten beeinflusst werden. Ferner wiesen Items mit offenem Format im Mittel eine höhere Itemschwierigkeit auf im Vergleich zu Items mit geschlossenem Format, wobei sich eine Interaktion zwischen Antwortformat und schriftsprachlichen Fähigkeiten andeutet. 764 H. Härtig et al. 3Schlüsselwörter Evaluation der Bildungsstandards · Kompetenztests · Schriftsprachliche Fähigkeiten · Antwortformate Analyses of the tasks for evaluating the educational standards in physics -Differentiation between written language proficiency and content knowledgeAbstract Within the evaluation of National Educational Standards in science education all items provide information required for the answer. The items thereby are also competency-based. We assume that general language proficiency plays a central role in order to assess achievement in physics since some answers require an open response. Thus, this paper addresses the question to what extent general language proficiency can affect the results of the evaluation of the educational standards in physics education for the competence content knowledge. Drawing on a sample of N = 1961 vocational trainees first, it is shown that the National Educational Standards items are affected by general language proficiency to a relevant degree. Furthermore, items in open response format are more difficult than items in a closed response format, with a slight interaction between response format and general language proficiency.

show abstract

A Comparison of Different Psychometric Approaches to Modeling Testlet Structures: An Example with C‐Tests

Cited by 16 publications

References 35 publications

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

Setting cut scores on an EFL placement test using the prototype group method: A receiver operating characteristic (ROC) analysis

A Motivational-Developmental Free Response Assessment Through a Bifactor Lens

Analyse der Aufgaben zur Evaluation der Bildungsstandards in Physik - Differenzierung von schriftsprachlichen Fähigkeiten und Fachlichkeit

Contact Info

Product

Resources

About