A Short Note on Aberrant Responses Bias in Item Response Theory

Jia, Bing Bing; Zhang, Xue; Zhu, Zhemin

doi:10.3389/fpsyg.2019.00043

Cited by 3 publications

(5 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Five hundred replications were conducted for each combination of factors. All factors have been previously found to be key factors for the measurement accuracy of diagnostic tests (Cui, Gierl, & Chang, 2012; Jia et al., 2019) as well as for the detection of person‐misfit relative to CDMs (Cui & Leighton, 2009).…”

Section: Simulation Studiesmentioning

confidence: 99%

“…Even the students themselves sometimes cannot tell whether a response to a certain item is normal or aberrant. Therefore, in practical applications, aberrant responses are simply defined as observed responses that do not match the expected ones (Meijer, 1996; Meijer, Muijtjens, & van der Vleuten, 1996; Meijer & Sijtsma, 2001; Jia, Zhang, & Zhu, 2019). The responses which do not match the expected ones can be obtained using simulations.…”

Section: Theories In Educationmentioning

confidence: 99%

“…There are seven different kinds of aberrant behaviour: high ability students' creative responding behaviour (Meijer, 1994), random guessing responses (Meijer, 1994), sleeping (Wright & Stone, 1979), fatigue (Petridou & Williams, 2007), aberrant guessing (Schuster & Yuan, 2011), random aberrant responses (Jia et al., 2019), and cheating (Meijer, 1994). These are the most common types of aberrant response behaviours from the existing literature.…”

Section: Machine Learningmentioning

confidence: 99%

“…Sleeping examinees can be simulated by generating item responses from high‐ability examinees with at most one missing attribute. In this simulation, the first 10 or 20% (Jia et al., 2019) of the items that the examinee ideally could get correct are imputed with incorrect responses to simulate sleeping behaviour. Cheating may happen in all kinds of exam.…”

Section: Simulation Studiesmentioning

confidence: 99%

See 3 more Smart Citations

A new person‐fit method based on machine learning in CDM in education

Zhu

Arthur

Chang

2022

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

Cognitive diagnosis models have become popular in educational assessment and are used to provide more individualized feedback about a student's specific strengths and weaknesses than traditional total scores. However, if the testing data are contaminated by certain biases or aberrant response patterns, such predictions may not be accurate. The current research objective is to develop a new person‐fit method that is based on machine learning and improves the functionality of existing person‐fit methods. Various simulations were designed under three aberrant conditions: cheating, sleeping and random guessing. Simulation results showed that the new method was more powerful and effective than previous methods, especially for short‐length tests.

show abstract

Section: Simulation Studiesmentioning

confidence: 99%

Section: Theories In Educationmentioning

confidence: 99%

Section: Machine Learningmentioning

confidence: 99%

Section: Simulation Studiesmentioning

confidence: 99%

See 2 more Smart Citations

A new person‐fit method based on machine learning in CDM in education

Zhu

Arthur

Chang

2022

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, IRT is highly sensitive to that kind of response disturbance since it is a statistical method based on an examinee's response to explain the ability level (Magis, 2014). The existence of aberrant responses may cause a strongly biased estimation of true underlying ability and may jeopardize the accuracy of measurements and invalidate the IRT use (Jia et al, 2019). Since CAT applications are also based on IRT models, they are also open to the same kind of biased estimations and erroneous measurements.…”

mentioning

confidence: 99%

The Effect of Aberrant Responses on Ability Estimation in Computer Adaptive Tests

GÖREN>

SARI

KARA>

et al. 2022

Eğitimde Ve Psikolojide Ölçme Ve Değerlendirme Dergisi

View full text Add to dashboard Cite

In computer adaptive testing (CAT), aberrant responses caused by some factors such as lucky guesses and carelessness errors may cause significant bias on ability estimation. Correct responses resulting from lucky guesses and false responses resulting from carelessness or anxiety may reveal aberrant responses and the impact of these types of aberrant responses may cause an erroneous estimation of the examinee’s actual ability because they do not reflect the examinee’s actual knowledge. In this study, the performances of 3PL and 4PL models regarding ability estimation were examined comparatively in the context of CAT simulations in case of aberrant responses. Under different conditions, twelve separate CAT simulations were conducted with 10 replications for each of the condition. Correlation, RMSE, bias, and mean absolute error (MAE) values were calculated and interpreted for each condition. Results generally indicated that the 4PL model provided more efficient and robust ability estimation than the 3 PL model and the 4PL model increased the precision and effectiveness of the CAT applications.

show abstract

Robustness of Computer Adaptive Tests to the Presence of Item Preknowledge: A Simulation Study

Kara,

Doğan,

Erdem Kara

2024

Eğitimde Ve Psikolojide Ölçme Ve Değerlendirme Dergisi

View full text Add to dashboard Cite

Item preknowledge describes a scenario where candidates may have access to some of the test items prior to the test administration. This involves sharing test materials and/or answers and it is difficult to identify the individuals with item preknowledge or the shared materials of the test. Nevertheless, it is essential to investigate the ‘item preknowledge’ problem because it can significantly affect the validity of the test results. It is believed that traditional linear tests are more robust to this type of aberrant response behavior than adaptive tests. In this context, the aim of this study is to examine the effect of item preknowledge on computer adaptive tests and identify the conditions under which adaptive tests are most resistant to the item preknowledge. For this purpose, a Monte Carlo simulation study was performed and 28 different conditions were examined. The results of the study indicated that the EAP estimation method provided better measurement precision than ML over all conditions. When 2PL and 3PL IRT models were compared, it was observed that 2PL had higher precision at most of the conditions. However, when the aberrancy ratio increased and reached 20% for both individuals and items, 3PL outperformed the 2PL model and gave the best results with the EAP combination. The results were discussed in line with the literature on item preknowledge and CAT and implications for practitioners and further research were provided.

show abstract

A Short Note on Aberrant Responses Bias in Item Response Theory

Cited by 3 publications

References 34 publications

A new person‐fit method based on machine learning in CDM in education

A new person‐fit method based on machine learning in CDM in education

The Effect of Aberrant Responses on Ability Estimation in Computer Adaptive Tests

Robustness of Computer Adaptive Tests to the Presence of Item Preknowledge: A Simulation Study

Contact Info

Product

Resources

About