Using Machine Learning Methods to Develop a Short Tree-Based Adaptive Classification Test: Case Study With a High-Dimensional Item Pool and Imbalanced Data

Abstract

This study explores advanced techniques in machine learning to develop a short tree-based adaptive classification test based on an existing lengthy instrument. A case study was carried out for an assessment of risk for juvenile delinquency. Two unique facts of this case are (a) the items in the original instrument measure a large number of distinctive constructs; (b) the target outcomes are of low prevalence, which renders imbalanced training data. Due to the high dimensionality of the items, traditional item response theory (IRT)-based adaptive testing approaches may not work well, whereas decision trees, which are developed in the machine learning discipline, present as a promising alternative solution for adaptive tests. A cross-validation study was carried out to compare eight tree-based adaptive test constructions with five benchmark methods using data from a sample of 3,975 subjects. The findings reveal that the best-performing tree-based adaptive tests yielded better classification accuracy than the benchmark method IRT scoring with optimal cutpoints, and yielded comparable or better classification accuracy than the best benchmark method, random forest with balanced sampling. The competitive classification accuracy of the tree-based adaptive tests also come with an over 30-fold reduction in the length of the instrument, only administering between 3 to 6 items to any individual. This study suggests that tree-based adaptive tests have an enormous potential when used to shorten instruments that measure a large variety of constructs.

Tomada de decisões baseada no parâmetro b da Teoria da Resposta ao Item nas classificações de ensemble

“…Os datasets desequilibrados inferem que a precisão não é um critério útil porque não reflete a precisão da classificação dos casos de classe minoritária. Entretanto para as classes majoritárias, classificadas corretamente, o valor da precisão é alto, mesmo que a classe minoritária seja classificada incorretamente (ZHENG; CHEON; KATZ, 2020).…”

A Methodological Review of Machine Learning in Applied Linguistics

“…Kumar and Boulanger (2020) introduced how deep learning can be applied in automatic essay scoring with the example of the Kaggle competition. Zheng et al (2020) took advantage of ML to develop an adaptive test system. In terms of cognitive diagnostic assessment (CDA), Almond et al (2007) introduced how the Bayesian networks can be applied to model CDA.…”

Designing A Shorter Form of the Big Three Perfectionism Scale: An Application of Ant Colony Optimization

“…Because many traditional measurement instruments can have numerous items and be time-consuming, the need for short and efficient measurement instruments has become a must for these types of studies. If the sample consists of children or youths, applying long scales is often unsuitable (Zheng et al, 2020). The BTPS (the focus of the current study) has several available short forms (e.g., Di Fabio et al, 2018; Feher et al, 2020).…”