Testing the Within-State Distribution in Mixture Models for Responses and Response Times

Kuijpers, Renske E.; Visser, Ingmar; Molenaar, Dylan

doi:10.3102/1076998620957240

Cited by 4 publications

(4 citation statements)

References 54 publications

(114 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Last, raters’ switches from normal behavior to aberrant scoring behavior may be dependent on the previous rating, and the current rating status may be influenced by the previous rating. A transition probability modeling approach ( Kuijpers et al, 2021 ) can be incorporated into the proposed models to describe the dependencies of behavior switching between adjacent rating sequences; this would be an interesting topic and is intended for future investigation.…”

Section: Discussionmentioning

confidence: 99%

Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments

Huang

2023

Applied Psychological Measurement

View full text Add to dashboard Cite

Rater effects are commonly observed in rater-mediated assessments. By using item response theory (IRT) modeling, raters can be treated as independent factors that function as instruments for measuring ratees. Most rater effects are static and can be addressed appropriately within an IRT framework, and a few models have been developed for dynamic rater effects. Operational rating projects often require human raters to continuously and repeatedly score ratees over a certain period, imposing a burden on the cognitive processing abilities and attention spans of raters that stems from judgment fatigue and thus affects the rating quality observed during the rating period. As a result, ratees’ scores may be influenced by the order in which they are graded by raters in a rating sequence, and the rating order effect should be considered in new IRT models. In this study, two types of many-faceted (MF)-IRT models are developed to account for such dynamic rater effects, which assume that rater severity can drift systematically or stochastically. The results obtained from two simulation studies indicate that the parameters of the newly developed models can be estimated satisfactorily using Bayesian estimation and that disregarding the rating order effect produces biased model structure and ratee proficiency parameter estimations. A creativity assessment is outlined to demonstrate the application of the new models and to investigate the consequences of failing to detect the possible rating order effect in a real rater-mediated evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments

Huang

2023

Applied Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…When they do not describe the processes in the respective states accurately, the selection criteria compensate for that by preferring a model with more states. Thus, there are not more latent states present in the data, but the submodels of the HMM are misspecified or too simple, potentially leading to spurious, extra, states being identified in the model selection process, see discussion and potential solutions in Kuijpers et al (2021). Correcting for model misspecifications led to a better model recovery in studies on animal movements (Langrock et al, 2015;Li & Bolker, 2017).…”

Section: Discussionmentioning

confidence: 99%

Characterising eye movement events with an unsupervised hidden markov model

Lüken

Kucharský²,

Visser³

2022

JEMR

Self Cite

View full text Add to dashboard Cite

Eye-tracking allows researchers to infer cognitive processes from eye movements that are classified into distinct events. Parsing the events is typically done by algorithms. Here we aim at developing an unsupervised, generative model that can be fitted to eye-movement data using maximum likelihood estimation. This approach allows hypothesis testing about fitted models, next to being a method for classification. We developed gazeHMM, an algorithm that uses a hidden Markov model as a generative model, has few critical parameters to be set by users, and does not require human coded data as input. The algorithm classifies gaze data into fixations, saccades, and optionally postsaccadic oscillations and smooth pursuits. We evaluated gazeHMM’s performance in a simulation study, showing that it successfully recovered hidden Markov model parameters and hidden states. Parameters were less well recovered when we included a smooth pursuit state and/or added even small noise to simulated data. We applied generative models with different numbers of events to benchmark data. Comparing them indicated that hidden Markov models with more events than expected had most likely generated the data. We also applied the full algorithm to benchmark data and assessed its similarity to human coding and other algorithms. For static stimuli, gazeHMM showed high similarity and outperformed other algorithms in this regard. For dynamic stimuli, gazeHMM tended to rapidly switch between fixations and smooth pursuits but still displayed higher similarity than most other algorithms. Concluding that gazeHMM can be used in practice, we recommend parsing smooth pursuits only for exploratory purposes. Future hidden Markov model algorithms could use covariates to better capture eye movement processes and explicitly model event durations to classify smooth pursuits more accurately.

show abstract

“…In real data analysis, practitioners should decide whether it is necessary to use the proposed iterative method. In order to have a deeper understanding of this method, future studies would compare the proposed method with other within-subject mixture approaches (e.g., Kuijpers et al, 2020; Molenaar et al, 2018) in various conditions. Fifth, as can be seen from the results, the proposed method may increase estimation error in the RT model, especially for time discrimination parameter.…”

Section: Discussionmentioning

confidence: 99%

Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process

Liu

2021

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

The prevalence and serious consequences of noneffortful responses from unmotivated examinees are well-known in educational measurement. In this study, we propose to apply an iterative purification process based on a response time residual method with fixed item parameter estimates to detect noneffortful responses. The proposed method is compared with the traditional residual method and noniterative method with fixed item parameters in two simulation studies in terms of noneffort detection accuracy and parameter recovery. The results show that when severity of noneffort is high, the proposed method leads to a much higher true positive rate with a small increase of false discovery rate. In addition, parameter estimation is significantly improved by the strategies of fixing item parameters and iteratively cleansing. These results suggest that the proposed method is a potential solution to reduce the impact of data contamination due to severe low test-taking effort and to obtain more accurate parameter estimates. An empirical study is also conducted to show the differences in the detection rate and parameter estimates among different approaches.

show abstract

Testing the Within-State Distribution in Mixture Models for Responses and Response Times

Cited by 4 publications

References 54 publications

Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments

Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments

Characterising eye movement events with an unsupervised hidden markov model

Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process

Contact Info

Product

Resources

About