A Probabilistic Filtering Approach to Non‐Effortful Responding

Abstract

Common response‐time‐based approaches for non‐effortful response behavior (NRB) in educational achievement tests filter responses that are associated with response times below some threshold. These approaches are, however, limited in that they require a binary decision on whether a response is classified as stemming from NRB; thus ignoring potential classification uncertainty in resulting parameter estimates. We developed a response‐time‐based probabilistic filtering procedure that overcomes this limitation. The procedure is rooted in the principles of multiple imputation. Instead of creating multiple plausible replacements of missing data, however, multiple data sets are created that represent plausible filtered response data. We propose two different approaches to filtering models, originating in different research traditions and conceptualizations of response‐time‐based identification of NRB. The first approach uses Gaussian mixture modeling to identify a response time subcomponent stemming from NRB. Plausible filtered data sets are created based on examinees' posterior probabilities of belonging to the NRB subcomponent. The second approach defines a plausible range of response time thresholds and creates plausible filtered data sets by drawing multiple response time thresholds from the defined range. We illustrate the workings of the proposed procedure as well as differences between the proposed filtering models based on both simulated data and empirical data from PISA 2018.

A screen-time-based mixture model for identifying and monitoring careless and insufficient effort responding in ecological momentary assessment data.

“…This can, for example, be achieved by using attentiveness probabilities as weights in the models applied to item responses for addressing substantive research questions. Doing so lowers the contribution of response patterns presumably stemming from C/IER to the estimation of parameters of interest and, as such, allows to obtain parameter estimates adjusted for presumed C/IER occurrence (see Ulitzsch, Domingue, et al, 2023; Ulitzsch, Shin, et al, 2023). Such adjustments could be part of an array of different sensitivity analyses, following recent calls to use multiverse analysis to evaluate the robustness of conclusions across different preprocessing scenarios for EMA data (Weermeijer et al, 2022).…”

“…To explore boundary conditions of applicability of the approach with the proposed component models, we varied the separation of attentive and careless screen times. Class separation is a well-studied, critical factor for obtaining trustworthy mixture model parameter estimates, with low-class separation challenging the accuracy of mixture model parameter estimates in general (Depaoli, 2012, 2013) and timing-based mixture modeling approaches for C/IER in particular (Pokropek, 2016; Ulitzsch, Domingue, et al, 2023). We, therefore, aimed to explore the degree of separation that is required for trustworthy parameter estimates for the proposed model.…”

Investigating dynamics in attentive and inattentive responding together with their contextual correlates using a novel mixture IRT model for intensive longitudinal data

“…When using such indicators, researchers have to decide on thresholds separating attentive behavior from C/IER. This decision is ultimately an arbitrary one, with even minor differences in threshold settings often heavily impacting conclusions on C/IER contamination (Niessen et al, 2016;Ulitzsch, Domingue, et al, 2023;Ulitzsch, Shin, & Lüdtke, 2023).…”

Open science perspectives on machine learning for the identification of careless responding: A new hope or phantom menace?

“…For a general discussion on such procedures for downweighting careless respondents, we refer to Ulitzsch et al. (2023a, 2023b). A disadvantage of this approach is that standard errors and hypothesis tests in the second step do not incorporate the uncertainty from obtaining the weights in the first step.…”

“…In the second step, these weights are applied in subsequent analysis so that careless respondents contribute less to the estimates. For a general discussion on such procedures for downweighting careless respondents, we refer to Ulitzsch et al (2023aUlitzsch et al ( , 2023b. A disadvantage of this approach is that standard errors and hypothesis tests in the second step do not incorporate the uncertainty from obtaining the weights in the first step.…”