Lying on the Dissection Table: Anatomizing Faked Responses

Abstract

Research has shown that even experts cannot detect faking above chance, but recent studies have suggested that machine learning may help in this endeavor. However, faking differs between faking conditions, previous efforts have not taken these differences into account, and faking indices have yet to be integrated into such approaches. We reanalyzed seven data sets (N = 1,039) with various faking conditions (high and low scores, different constructs, naïve and informed faking, faking with and without practice, different measures [self-reports vs. implicit association tests; IATs]). We investigated the extent to which and how machine learning classifiers could detect faking under these conditions and compared different input data (response patterns, scores, faking indices) and different classifiers (logistic regression, random forest, XGBoost). We also explored the features that classifiers used for detection. Our results show that machine learning has the potential to detect faking, but detection success varies between conditions from chance levels to 100%. There were differences in detection (e.g., detecting low-score faking was better than detecting high-score faking). For self-reports, response patterns and scores were comparable with regard to faking detection, whereas for IATs, faking indices and response patterns were superior to scores. Logistic regression and random forest worked about equally well and outperformed XGBoost. In most cases, classifiers used more than one feature (faking occurred over different pathways), and the features varied in their relevance. Our research supports the assumption of different faking processes and explains why detecting faking is a complex endeavor.

Can People With Higher Versus Lower Scores on Impression Management or Self-Monitoring Be Identified Through Different Traces Under Faking?

“…Although a detailed and complete list of them is beyond the scope of this article, in the following, we will nevertheless provide an overview of the most prominent traces that have been used in faking research along with prominent approaches in faking detection. So far, none of these procedures have been widely accepted ( Röhner et al, 2022 ). Also, none of them work without misclassification, which indicates that there are still factors that are not understood in the detection of faking.…”

“…This phenomenon is called intermittent faking ( Brown & Böckenholt, 2022 ). In this line of thinking, recent studies have demonstrated that test scores are not able to capture the complete picture of faking (e.g., Calanna et al, 2020 ; Röhner et al, 2022 ). Relatedly, studies have shown that looking at individuals’ responses to each individual item on the to-be-faked measure (in its entirety called individuals’ response patterns ; e.g., Calanna et al, 2020 ) may offer better insights into faking because individual items include information about traces of faking that are washed out in test scores (e.g., Böckenholt, 2013 , 2017 ; Calanna et al, 2020 ; Lee et al, 2022 ; Röhner et al, 2022 ; Sun et al, 2022 ).…”

“…In this line of research, by using item response theory (IRT) models, studies have demonstrated that faking detection is improved by dissecting individual item responses into item response trees and using the item response trees in modeling (e.g., Böckenholt, 2013 , 2017 ; Lee et al, 2022 ; Sun et al, 2022 ). Relatedly, research using machine learning and item responses has successfully detected faking with high levels of accuracy (e.g., Calanna et al, 2020 ; Röhner et al, 2022 ). Taken together, the results point to the necessity of investigating responses on individual items as the more reliable approach for tracing faking.…”

On the Death of Implicit Association Tests (IATs)

“…On the basis of these claims, the definition of IATs (Greenwald & Banaji, 1995) should probably be reworked. Similarly, the claim that IAT effects result from uncontrollable processes (for an introduction, see De Houwer, 2006) has been empirically disproved by research showing that participants can fake on IATs (e.g., Röhner et al, 2022). This lack of clarity about what is implicit in IATs can be viewed as the first disease that has contributed to the death of IATs.…”

“…With respect to the fakeability of IATs, researchers have developed several faking indices (e.g., Röhner et al, 2023). However, research using machine learning demonstrated that faking indices were outperformed by processes such as participants’ speed-accuracy tradeoff that were disentangled with diffusion models to detect faking (Röhner et al, 2022). These results reflect theory and earlier empirical findings on faking in IATs that have been revealed to involve deliberately adapting speed and accuracy (e.g., Röhner et al., 2013).…”