The Influence of Test Characteristics on the Detection of Aberrant Response Patterns

Abstract

Statistical methods to assess the congruence between an item response pattern and a specified item response theory model have recently proliferated. This "person fit" research has focused on the question: To what extent can person-fit indices identify well-defined forms of aberrant item response? This study extended previous person-fit research in two ways. First, an unexplored model for generating aberrant response patterns was explicated. The data-generation model is based on the theory that aberrant item responses result in less psychometric information for the individual than predicated by the parameters of a specified response model. Second, the proposed response aberrancy generation model was implemented to investigate how the aberrancy detection power of a person-fit statistic is influenced by test properties (e.g., the spread of item difficulties). Results indicated that detecting aberrant response patterns was especially problematic for tests with less than 20 items, and for tests with limited ranges of item difficulty. An applied consequence of these results is that certain types of test designs (e.g., peaked tests) and administration procedures (e.g., adaptive tests) potentially act to limit the detection of aberrant item responses.

Seriousness checks are useful to improve data validity in online research

“…However, the applicability of such tests is limited. They can be employed only if a test was constructed to fit an item response model (Li & Olejnik, 1997;van den Wittenboer, Hox, & de Leeuw, 1997), and their power highly depends on the length of a test and the range of the difficulties of its items (Reise & Due, 1991).…”

Detection of Person Misfit in Computerized Adaptive Tests with Polytomous Items

“…By dichotomizing the scores the length of the tests is actually decreased, which is unfavorable for the assessment of person fit (Reise & Due, 1991). Also, compared with the dichotomized version of the item, the item information function of a polytomously scored item is higher at the peak of the function and the information is also distributed across a wider range of 0, which may also enhance the assessment of person fit (Reise & Due, 1991).…”

“…By dichotomizing the scores the length of the tests is actually decreased, which is unfavorable for the assessment of person fit (Reise & Due, 1991). Also, compared with the dichotomized version of the item, the item information function of a polytomously scored item is higher at the peak of the function and the information is also distributed across a wider range of 0, which may also enhance the assessment of person fit (Reise & Due, 1991). Finally, the dichotomization that is chosen for a polytomous score variable has a substantial effect on the measurement outcome 0, unless specific conditions on the item parameters hold (Jansen & Roskam, 1986, Roskam & Jansen, 1989).…”

Application de l’indice lz pour l’élimination de données de recherche en langues

“…À l'opposé, une valeur positive élevée démontre que le patron de réponses observé est tout simplement plus probable que le patron de réponses attendu : une situation qui n'est pas associée à un patron de réponses inapproprié. les raisons principales qui ont motivé le choix de l z résident dans le fait qu'il est reconnu comme l'un des plus puissants et populaires indices statistiques pour détecter les patrons de réponses inappropriés (drasgow & levine, 1986 ;li & olejnik, 1997 ;nering, 1997 ;nering & Meijer, 1998 ;raîche, Magis, Béland & Blais, 2011 ;raîche, Magis, Blais & Brochu, 2013 ;reise & due, 1991). Par contre, il est important de signaler que cet indice permet seulement de détecter si un patron de réponses est approprié ou inapproprié.…”