Effects of Calibration Sample Size and Item Bank Size on Ability Estimation in Computerized Adaptive Testing

Şahin, Alper; Weiss, David J.

doi:10.12738/estp.2015.6.0102

Cited by 6 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, in some cases the complexity associated with administering and scoring an adaptive test may not be warranted given similar performance can be achieved through the use of a static short form comparator test (Reise & Henson, 2000). Indeed, studies have shown that the efficiency and precision associated with different CATs can vary depending on the quality of the item bank and the number of items that target specific severity levels on the latent traits (Şahin & Weiss, 2015). The current study aimed to determine the efficiency versus precision of an adaptive test based on multidimensional IRT through the use of computer simulations.…”

Section: The Current Studymentioning

confidence: 99%

Developing and Validating a Computerized Adaptive Test to Measure Broad and Specific Factors of Internalizing in a Community Sample

et al. 2017

View full text Add to dashboard Cite

Highly efficient assessments that better account for comorbidity between mood and anxiety disorders (internalizing) are required to identify individuals who are most at risk of psychopathology in the community. The current study examined the efficiency and validity associated with a multidimensional computerized adaptive test (CAT) to measure broad and specific levels of internalizing psychopathology. The sample comprised 3,175 respondents to an online survey. Items from five banks (generalized anxiety, depression, obsessive-compulsive disorder, panic disorder, social anxiety disorder) were jointly calibrated using a bifactor item response theory model. Simulations indicated that an adaptive algorithm could accurately ( rs ≥ 0.90) estimate general internalizing and specific disorder scores using on average 44 items in comparison with the full 133-item bank (67% reduction in items). Scores on the CAT demonstrate convergent and divergent validity with previously validated short severity scales and could significantly differentiate cases of DSM-5 disorder. As such, the CAT validly measures both broad and specific constructs of internalizing disorders in a manner similar to the full item bank and a static brief form but with greater gains in efficiency and, therefore, a reduced degree of respondent burden.

show abstract

Section: The Current Studymentioning

confidence: 99%

Developing and Validating a Computerized Adaptive Test to Measure Broad and Specific Factors of Internalizing in a Community Sample

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Furthermore, CATs with a relatively smaller number of items in their item banks have been shown to function suitably. Sahin and Weiss (2015) have demonstrated that small item banks can still lead to accurate item parameter and ability estimates in CAT using simulated data with items having sufficient information. In another study conducted on a language test using real data, CATs with as few as 20 items were shown to produce promising results with SE values of 0.35 (Kaya & Kalender, 2018).…”

Section: Instrumentmentioning

confidence: 99%

IRT-based classification analysis of an English language reading proficiency subtest

2022

View full text Add to dashboard Cite

Language proficiency testing serves an important function of classifying examinees into different categories of ability. However, misclassification is to some extent inevitable and may have important consequences for stakeholders. Recent research suggests that classification efficacy may be enhanced substantially using computerized adaptive testing (CAT). Using real data simulations, the current study investigated the classification performance of CAT on the reading section of an English language proficiency test and made comparisons with the paper-based version of the same test. Classification analysis was carried out to estimate classification accuracy (CA) and classification consistency (CC) by applying different locations and numbers of cutoff points. The results showed that classification was suitable when a single cutoff score was used, particularly for high- and low-ability test takers. Classification performance declined significantly when multiple cutoff points were simultaneously employed. Content analysis also raised important questions about construct coverage in CAT. The results highlight the potential for CAT to serve classification purposes and outline avenues for further research.

show abstract

“…Within each interval, 25 realizations of a uniform distribution with the corresponding bounds were generated (Finkelman et al, 2010). The c parameters were realizations of a U[0.00, 0.25] distribution (Sahin & Weiss, 2015). In this EF item bank consisting of the 300 true (i.e., not estimated) item parameters, the means and standard deviations for the a, b, and c parameters were 1.21 (0.40), 0.00 (1.73), and 0.12 (0.07), respectively.…”

Section: Simulationmentioning

confidence: 99%

Robustness of Adaptive Measurement of Change to Item Parameter Estimation Error

Cooperman

Weiss

Wang

2021

Educational and Psychological Measurement

View full text Add to dashboard Cite

Adaptive measurement of change (AMC) is a psychometric method for measuring intra-individual change on one or more latent traits across testing occasions. Three hypothesis tests—a Z test, likelihood ratio test, and score ratio index—have demonstrated desirable statistical properties in this context, including low false positive rates and high true positive rates. However, the extant AMC research has assumed that the item parameter values in the simulated item banks were devoid of estimation error. This assumption is unrealistic for applied testing settings, where item parameters are estimated from a calibration sample before test administration. Using Monte Carlo simulation, this study evaluated the robustness of the common AMC hypothesis tests to the presence of item parameter estimation error when measuring omnibus change across four testing occasions. Results indicated that item parameter estimation error had at most a small effect on false positive rates and latent trait change recovery, and these effects were largely explained by the computerized adaptive testing item bank information functions. Differences in AMC performance as a function of item parameter estimation error and choice of hypothesis test were generally limited to simulees with particularly low or high latent trait values, where the item bank provided relatively lower information. These simulations highlight how AMC can accurately measure intra-individual change in the presence of item parameter estimation error when paired with an informative item bank. Limitations and future directions for AMC research are discussed.

show abstract

Effects of Calibration Sample Size and Item Bank Size on Ability Estimation in Computerized Adaptive Testing

Cited by 6 publications

References 16 publications

Developing and Validating a Computerized Adaptive Test to Measure Broad and Specific Factors of Internalizing in a Community Sample

Developing and Validating a Computerized Adaptive Test to Measure Broad and Specific Factors of Internalizing in a Community Sample

IRT-based classification analysis of an English language reading proficiency subtest

Robustness of Adaptive Measurement of Change to Item Parameter Estimation Error

Contact Info

Product

Resources

About