A Multidimensional IRT Approach for Dynamically Monitoring Ability Growth in Computerized Practice Environments

Abstract

Adaptive learning systems have received an increasing attention as they enable to provide personalized instructions tailored to the behaviors and needs of individual learners. In order to reach this goal, it is desired to have an assessment system, monitoring each learner's ability change in real time. The Elo Rating System (ERS), a popular scoring algorithm for paired competitions, has recently been considered as a fast and flexible method that can assess learning progress in online learning environments. However, it has been argued that a standard ERS may be problematic due to the multidimensional nature of the abilities embedded in learning materials. In order to handle this issue, we propose a system that incorporates a multidimensional item response theory model (MIRT) in the ERS. The basic idea is that instead of updating a single ability parameter from the Rasch model, our method allows a simultaneous update of multiple ability parameters based on a compensatory MIRT model, resulting in a multidimensional extension of the ERS (“M-ERS”). To evaluate the approach, three simulation studies were conducted. Results suggest that the ERS that incorrectly assumes unidimensionality has a seriously lower prediction accuracy compared to the M-ERS. Accounting for both speed and accuracy in M-ERS is shown to perform better than using accuracy data only. An application further illustrates the method using real-life data from a popular educational platform for exercising math skills.

TriMElo: Uma Proposta de Modelo para Avaliação Multi-Habilidades

“…O uso do sistema de classificação Elo oferece vantagens, tais como simplicidade de uso em ambientes online e implementação em sistemas educacionais, além de apresentar baixo quantitativo de parâmetros para ajustar [Pelánek 2016]. Todavia, destina-se a rastrear apenas umaúnica habilidade [Park et al 2019].…”

“…Na educação, o Elo estabelece que um estudanteé considerado um jogador e o problemaé considerado seu adversário [Pelánek 2016]. No entanto tem sido argumentado que o Elo pode ser problemático devidoà natureza das habilidades incorporadas nos materiais de aprendizagem, pois destina-se a rastrear apenas umaúnica habilidade [Park et al 2019].…”

“…O modelo M-ERS proposta por [Park et al 2019], utiliza um modelo TRIM Compensatório e o modelo Elo para estimar as múltiplas habilidades dos estudantes em sistemas de aprendizagem adaptativa, assumindo que um item pode envolver mais de uma habilidade e que uma baixa habilidade pode ser compensada por outra mais alta. Dessa forma, no presente trabalho, definiu-se habilidade como a habilidade geral em programação, e sub-habilidades como as múltiplas habilidades utilizadas na elaboração da solução.…”

A Comparison between BMIRT and IRTPRO: A Simulation Study of a Multidimensional Item Response Model

“…MIRT models are considered by a variety of practitioners working on competency assessment, patient-reported outcome measurements, and psychological inventory [19,20]. In this study, the main focus is only on how MIRT is used in competency assessment.…”

Harmonized Phenotypes for Anxiety, Depression, and Attention-Deficit Hyperactivity Disorder (ADHD)

“…There are multiple ways to carry out test linking and one of the widely used approaches is the framework of Item Response Theory (IRT; Embretson & Reise, 2000;Kolen & Brennan, 2014). The use of IRT is a common and flexible method for modeling the relationship between participants` trait levels and their responses to items (Park et al, 2019;van den Berg et al, 2007van den Berg et al, , 2014. In the IRT approach, a participant`s response to an item depends on both the participant`s trait level and the item parameters of that particular item (Embretson & Reise, 2000).…”