Assessing Testlet Effect, Impact, Differential Testlet, and Item Functioning Using Cross-Classified Multilevel Measurement Modeling

Ravand, Hamdollah

doi:10.1177/2158244015585607

Cited by 6 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An important assumption in assessing the dimensionality of a test within the latent variable modeling framework is local independence, whether in terms of local item independence or of local person independence (Jiao et al, 2012; Reckase, 2009). Local item (or person) independence assumptions fail to hold when there is systematic grouping of items (or persons), as this clustering creates similarities among items (or persons) that cannot be attributed to the latent traits being measured (Cho et al, 2014; Jiao et al, 2012; Ravand, 2015). Features such as test multidimensionality or multilevel data present a challenge to local independence.…”

mentioning

confidence: 99%

Resolving Dimensionality in a Child Assessment Tool: An Application of the Multilevel Bifactor Model

Akaeze

Lawrence

2022

Educational and Psychological Measurement

View full text Add to dashboard Cite

Multidimensionality and hierarchical data structure are common in assessment data. These design features, if not accounted for, can threaten the validity of the results and inferences generated from factor analysis, a method frequently employed to assess test dimensionality. In this article, we describe and demonstrate the application of the multilevel bifactor model to address these features in examining test dimensionality. The tool for this exposition is the Child Observation Record Advantage 1.5 (COR-Adv1.5), a child assessment instrument widely used in Head Start programs. Previous studies on this assessment tool reported highly correlated factors and did not account for the nesting of children in classrooms. Results from this study show how the flexibility of the multilevel bifactor model, together with useful model-based statistics, can be harnessed to judge the dimensionality of a test instrument and inform the interpretability of the associated factor scores.

show abstract

mentioning

confidence: 99%

Resolving Dimensionality in a Child Assessment Tool: An Application of the Multilevel Bifactor Model

Akaeze

Lawrence

2022

Educational and Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…In the literature on testlet, a number of studies were documented in which the item parameters of testlets were estimated and the reliability of the scores obtained from them was examined by employing the above-mentioned different approaches (Eckes, 2014; Eckes & Baghaei, 2015; Li, Li et al, 2010; Paap et al, 2015; Ravand, 2015; Shaw et al, 2020). Also, these issues have been addressed in many studies within the scope of GT (Chien, 2008; Kaya Uyanık & Gelbal, 2018; Lee, 2000; Lee & Frisbie, 1999; Lee & Park, 2012).…”

Section: Introductionmentioning

confidence: 99%

Examination of Testlet Effect in Open-Ended Items

Uyanık

Ertuna

2022

Sage Open

View full text Add to dashboard Cite

The testlets are prevalently used in national and international standard tests and the teacher-made achievement tests thanks to their practicality. The purpose of the current study is to evaluate the testlet effect in test consisting the open-ended items and to identify the possible errors when this effect is ignored. The data consists of the scores given by three raters of the responses of 380 students in the teacher training program to two open-ended items in two testlets. The two designs with and without the testlet effect were modeled (totally crossed random design and the nested random design) and the differences between them were analyzed by Generalizability Theory (GT). The results showed that raters had a slight effect on the scores in both models tested, and the reliability of the scores was overestimated when the testlet effect was not ignored in such measurement tools.

show abstract

“…The solutions to addressing local item dependence are as follows: 1) using score-based polytomous IRT models such as the graded response model and polytomous logistic regression and 2) using the testlet-effects models. Score-based approaches have been criticized for reporting biased parameter estimates and substantial overestimation of reliability and test information values (Ravand, 2015). The application of testlet-effects models, namely Bifactor and Testlet Response Theory (TRT) models, are preferred in order to account for LID without loss of information.…”

Section: Testlet Effectmentioning

confidence: 99%

Measurement of Reading Literacy, Growth, and Learning Potential of Grade 9 Students: Application of Computerized Dynamic Assessment Concept

Lunrasri¹

View full text Add to dashboard Cite

The aims of this research were to (1) develop and validate the computerized dynamic assessment for reading literacy and (2) study the effects of different types of prompting of the computerized dynamic assessment on reading literacy, growth of learning, and learning potential. The pilot samples were a total of 802 ninth-grade students in six secondary schools in Bangkok. The data for the main study was gathered from 541 ninth-grade students in eleven secondary schools in Bangkok. A quasi-experimental design was adopted. Research instruments were reading literacy tests for computerized dynamic assessment. The data were analyzed as follows: 1) ANCOVA statistics for reading literacy performance, 2) latent growth modeling for growth of learning, and 3) mixed ANOVA and MANOVA statistics for learning potential. Findings were presented as follows: 1. This research developed computerized dynamic assessment for reading literacy, which was interactive and online feedback program. The tested contents were composed of three dimensions of reading literacy, including 1) locate information, 2) understand, and 3) evaluate and reflect. Three parallel tests were assessed at three different time points, each test comprised twenty two-tier items. For the psychometric properties of the instrument, most items for all reading literacy test forms were appropriate for content validity and parallelism. For the model comparison, the bifactor MIRT model was the best-fitting model. A majority of the items had good multidimensional discrimination values. The multidimensional difficulty estimates were in the acceptable range for most items. Moreover, the instruments yielded highly internal reliability. Regarding the statistical parallelism of the test forms, they showed satisfactory conformity at the test forms and item-by-item levels. 2. For the results of reading literacy performance, different types of promptings of computerized dynamic assessment had a significant effect on students’ reading posttest scores. Verification prompting group received significantly lower posttest score than other prompting-based groups. For the growth of learning, mixed prompting group obtained the highest rate of growth in reading literacy than other groups. For the results of the associations of reading literacy subscales, the growth rate in one subscale did not relate to the growth rate in other subscales, except mixed prompting group that the growth rate in understand was associated with the growth rate in evaluate and reflect. In terms of learning potential, the results of the availability score revealed that there were significant differences in availability scores measured at the third testing session among groups with different types of promptings. Verification prompting group had a significantly higher availability score when compared with mixed prompting group. For the mediated score, the result showed that there were no significant two-way interactions between prompting conditions and time on the mediated score. However, there was a significant main effect of prompting conditions. Verification prompting group had a significantly lower mediated score when compared with other groups. For the levels of prompting, the result showed that there were significant differences in first level of prompting, second level of prompting, third level of prompting, and fourth level of prompting among groups with different types of promptings. Verification prompting obtained significantly higher assistance than mixed prompting group in all levels of prompting.

show abstract

Assessing Testlet Effect, Impact, Differential Testlet, and Item Functioning Using Cross-Classified Multilevel Measurement Modeling

Cited by 6 publications

References 26 publications

Resolving Dimensionality in a Child Assessment Tool: An Application of the Multilevel Bifactor Model

Resolving Dimensionality in a Child Assessment Tool: An Application of the Multilevel Bifactor Model

Examination of Testlet Effect in Open-Ended Items

Measurement of Reading Literacy, Growth, and Learning Potential of Grade 9 Students: Application of Computerized Dynamic Assessment Concept

Contact Info

Product

Resources

About