Detecting computer-generated random responding in online questionnaires: An extension of Dupuis, Meier &amp; Cuneo (2019) on dichotomous data

Dupuis, Marc; Meier, Emanuele; Gholam‐Rezaee, Mehdi; Gmel, Gerhard; Strippoli, Marie‐Pierre F.; Renaud, Olivier

doi:10.1016/j.paid.2020.109812

Cited by 10 publications

(19 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We consider four NRIs whose ROC curves had impressive (e.g., upwards of 95%) area under the curve (AUC) in Dupuis et al (2019). Of the four considered NRIs, Mahalanobis distance and person-total correlation are widely used in the literature (Curran, 2016;DeSimone & Harms, 2018;Dupuis et al, 2019Dupuis et al, , 2020Meade & Craig, 2012;Niessen et al, 2016). Let z be a respondent vector of raw Likert responses (i.e., z tr i or z i ).…”

Section: Common To Variousmentioning

confidence: 99%

“…Two more NRIs were taken from functional method theory (FMT), response coherence and response reliability (Dupuis, Meier, Capel, & Gendre, 2015;Dupuis et al, 2019Dupuis et al, , 2020. FMT is a testing framework competing with classical test theory and item response theory (Dupuis et al, 2015).…”

Section: Common To Variousmentioning

confidence: 99%

“…The package mvtnorm (Genz et al, 2021) was used to compute multivariate Gaussian densities. FMT computations were our own implementation based on Dupuis et al (2020). For parallel processing, packages furrr (Vaughan & Dancho, 2021) and future (Bengtsson, 2021) were used.…”

Section: Compute Outcome Measuresmentioning

confidence: 99%

“…The premise underlying use of NRIs is that in NRI space, aberrant responders form a different cluster from non-abberant ones (Dupuis et al, 2019(Dupuis et al, , 2020. To illustrate, Figure 1 is a plot of Mahalanobis distance and person-total correlation for a sample of humans and bots, n human = 50 and n bot = 50, on the Humor Styles Questionnaire (HSQ) (Kuiper, 2016).…”

mentioning

confidence: 99%

“…Anecdotally, researchers may choose arbitrary cutoff values based on experience; if done post-hoc, this could constitute use of a researcher degree of freedom (Simmons, Nelson, & Simonsohn, 2011). Some studies go only as far as to show that their chosen NRIs discriminate well between the two classes, not how to find optimal cutoffs (Dupuis et al, 2019(Dupuis et al, , 2020Meade & Craig, 2012). In Meade and Craig (2012), Mahalanobis distance was shown to be a significant predictor of class in the calibration sample.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Supervised classes, unsupervised mixing proportions: Detection of bots in a Likert-type questionnaire

Ilagan¹,

Falk²

2022

Preprint

View full text Add to dashboard Cite

Administering Likert-type questionnaires to online samples risks contamination of the data by malicious computer-generated random responses, i.e., bots. Although nonresponsivity indices (NRIs) such as person-total correlations or Mahalanobis distance have shown great promise to detect bots, universal cutoff values are elusive. An initial calibration sample constructed via stratified sampling of bots and humans---real or simulated under a measurement model---has been used to empirically choose cutoffs with a high nominal specificity. However, a high-specificity cutoff is less accurate when the target sample has a high contamination rate. In the present article, we propose the supervised classes, unsupervised mixing proportions (SCUMP) algorithm that chooses a cutoff to maximize accuracy. SCUMP uses a Gaussian mixture model to estimate, unsupervised, the contamination rate in the sample of interest. A simulation study found that, in the absence of model misspecification on the bots, our cutoffs maintained accuracy across varying contamination rates.

show abstract