Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting

Abstract

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

Careless responding in crowdsourced alcohol research: A systematic review and meta-analysis of practices and prevalence.

“…Our pooled prevalence estimates of ∼12% of individuals who could be categorized as careless responders is similar to estimates across different samples and fields of study (between 10% and 15%; Goldammer et al, 2020; Schroeders et al, 2021). These comparisons suggest, at least for the studies we were able to identify, that careless responding is no more prevalent in alcohol-related research than other fields.…”

“…Nonmutually exclusive might be a lack of attentive responses or failure to read and follow instructions, which can be identified by “attention checks” (Göritz et al, 2021). Importantly for the researcher, poor quality data can increase noise (nonrandom error variance), making it much more difficult to ascertain any signal within the noise (Schroeders et al, 2021). Furthermore, if left unaccounted for, careless responders “pose a great threat to replicability” (Curran, 2016, p. 5).…”

Validation and generalizability of machine learning prediction models on attrition in longitudinal studies

“…GBM sequentially combine multiple single decision trees that usually have a comparably poor predictive accuracy (Breiman, 2001). One advantage of GBM is that researchers do not have to a priori parameterize the relationship between an outcome and its predictors, which makes them popular for supervised classification tasks (e.g., Schroeders et al, 2022).…”

The ephemeral nature of wording effects

“…On the other hand, several authors have proposed different approaches to detect careless responses with scales that contain or do not contain reverse items, including attentional evaluation techniques (bogus or instructed response items; Meade & Craig, 2012;Oppenheimer et al, 2009), evaluation of auxiliary data or paradata (response latencies or fixations; Henninger & Plieninger, 2021;Koutsogiorgi & Michaelides, 2022;Zhang & Conrad, 2014) and detection of outliers or response functions (Curran et al, 2016). However, the evidence shows a series of drawbacks associated with the use and interpretation of these techniques, including the lack of effectiveness and consistency between them due to the arbitrariness with which their cutoff points are defined, depending on the dataset used (Curran, 2016;Nissen et al, 2016) and low sensitivity and specificity for the detection of not completely random response patterns (Meade et al, 2017;Schroeders et al, 2022).…”