Comparing countdown- and IRT-based approaches to computerized adaptive personality testing.

Abstract

Computerized adaptive testing (CAT) is an emerging technology in the personality assessment literature given the greater efficiency it affords compared with traditional methods. However, few studies have directly compared the efficiency and validity of 2 competing methods for personality CAT: (a) methods based on item response theory (IRT-CAT) versus (b) methods based on the countdown method (CM-CAT). To that end, we conducted real-data simulations with previously collected responses (N = 8,690) to the Schedule for Nonadaptive and Adaptive Personality (SNAP). Three CAT algorithms (IRT-CAT, IRT-CAT with 5-item minimum, CM-CAT) were evaluated for item savings, classification accuracy, and convergent/discriminant validity. All CATs yielded lower classification accuracy and validity than traditional testing but required 18%-86% fewer items. Ultimately, the IRT-CAT, with minimum 5-item requirement, struck the most ideal balance of highest item savings, and generally fewer costs to validity and accuracy. These results confirm findings regarding item savings trends from previous CAT studies. In addition, this study provides a model for how the validity and precision of CATs may be compared across personality assessments.

Cross-validation of short forms of the Screener and Opioid Assessment for Patients with Pain-Revised (SOAPP-R)

“…Therefore, the average amount of time spent on the full-length SOAPP-R could not be computed, nor could the amount of time that would be saved by administering a shorter version. Reduction in test length itself is a commonly used measure of the savings provided by shortened instruments (e.g., Finkelman et al, 2015;Leidy and Knebel, 2010;Rudick et al, 2013;Smits et al, 2011;Thompson, 2007Thompson, , 2011; nevertheless, future studies should record participants' response times so that time savings can be calculated as well. Even a small time savings for some individual patients would cumulatively reduce administrative burden.…”

Statistical measurement of trees’ similarity

“…We chose Kappa (ƙ) because it is the observed proportion of agreement between the assigners after chance agreement is removed from consideration (Cohen 1968). Kappa (ƙ) is widely used as a measure of association for contingency tables (Hambleton and Zaal 2013;Rudick et al 2013;Sengupta and Te'eni, D. 1993;You et al 2012). In addition, Landis and Koch (1977) proposed the English-language meanings of Kappa (ƙ) thresholds featured in Table 1.…”

Item Banking for an Adaptive Measurement of Neuroticism

“…Further research will also seek to validate a cut-off on the Neuroticism scale in order to differentiate subjects with clinically significant levels. The study of an interruption modality based on clinical criteria can optimize adaptive measurement if it is intended to be used for evaluation tasks with screening purposes (Fonseca-Pedrero et al, 2013, Rudick et al, 2013. In this circumstance, administration may be briefer, because items that showed maximum discrimination capacity at the level of the trait associated with the cut-off point, are only selected.…”