With the improvement of technologies, item preknowledge has become a common concern in the field of test security. The present study proposes a machine-learning-based approach to detect compromised items and examinees with item preknowledge simultaneously in computerized adaptive testing. Drawing on ideas in ensemble learning, this detection approach samples multiple subsets from the original response data, conducts sub-detections independently for each subset, and combines all sub-detection results into one detection result. Each sub-detection is a semi-supervised learning, running the following four steps iteratively until stop criteria are met: (1) select training samples and train a classification model; (2) select testing samples and predict the classes of the samples; (3) identify questionable examinees and questionable items based on the prediction result; (4) update the data for the next iteration. The experiment shows that under the conditions studied, provided the amount of preknowledge is not overwhelming, the approach controls the false negative rate at a relatively low level and the false positive rate at a very low level.

Discrete-option multiple-choice (DOMC) items differ from traditional multiple-choice (MC) items in the sequential administration of response options (up to display of the correct option). DOMC can be appealing in computer-based test administrations due to its protection of item security and its potential to reduce testwiseness effects. A psychometric model for DOMC items that attends to the random positioning of key location across different administrations of the same item is proposed, a feature that has been shown to affect DOMC item difficulty. Using two empirical data sets having items administered in both DOMC and MC formats, the variability in key location effects across both items and persons is considered. The proposed model exploits the capacity of the DOMC format to isolate both (a) distinct sources of item difficulty (i.e., related to the identification of keyed responses versus the ruling out of distractor options) and (b) distinct person proficiencies related to the same two components. Practical implications in terms of the randomized process applied to schedule item key location in DOMC test administrations are considered.

No abstract

As technologies have been improved, item preknowledge has become a common concern in the test security area. The present study proposes an unsupervised‐learning‐based approach to detect compromised items. The unsupervised‐learning‐based compromised item detection approach contains three steps: (1) classify responses of each examinee as either normal or aberrant based on both the item response and the response time; (2) use a recursive algorithm to cluster examinees into groups based on their response similarity; (3) identify the group with strongest preknowledge signal and report questionable items as compromised. Results show that under the conditions studied, provided the amount of preknowledge is not overwhelming and aberrance effect is at least moderate, the approach controls the false‐negative rate at a relatively low level and the false‐positive rate at an extremely low level.

Pan and Wollack (PW) proposed a machine learning method to detect compromised items. We extend the work of PW to an approach detecting compromised items and examinees with item preknowledge simultaneously and draw on ideas in ensemble learning to relax several limitations in the work of PW. The suggested approach also provides a confidence score, which is based on an autoencoder to represent our confidence that the detection result truly corresponds to item preknowledge. Simulation studies indicate that the proposed approach performs well in the detection of item preknowledge, and the confidence score can provide helpful information for users.