Interest in unintended discrimination that can result from implicit attitudes and stereotypes (implicit biases) has stimulated many research investigations. Much of this research has used the Implicit Association Test (IAT) to measure association strengths that are presumed to underlie implicit biases. It had been more than a decade since the last published treatment of recommended best practices for research using IAT measures. After an initial draft by the first author, and continuing through three subsequent drafts, the 22 authors and 14 commenters contributed extensively to refining the selection and description of recommendation-worthy research practices. Individual judgments of agreement or disagreement were provided by 29 of the 36 authors and commenters. Of the 21 recommended practices for conducting research with IAT measures presented in this article, all but two were endorsed by 90% or more of those who felt knowledgeable enough to express agreement or disagreement; only 4% of the totality of judgments expressed disagreement. For two practices that were retained despite more than two judgments of disagreement (four for one, five for the other), the bases for those disagreements are described in presenting the recommendations. The article additionally provides recommendations for how to report procedures of IAT measures in empirical articles.

Flake, Pek, and Hehman (2017) recently demonstrated that metrics of structural validity are severely underreported in social and personality psychology. We apply their recommendations for the comprehensive assessment of structural validity to a uniquely large and varied dataset (N = 144496 experimental sessions) to investigate the psychometric properties of some of the most widely used self-report measures (k = 15 questionnaires, 26 subscales) in social and personality psychology. When assessed using the modal practice of considering only their internal consistency, 89% of scales appeared to possess good validity. Yet, when validity was assessed comprehensively (via internal consistency, immediate and delayed test-retest reliability, factor structure, and measurement invariance for median age and gender) only 4% demonstrated good validity. Furthermore, the less commonly a test is reported in the literature, the more likely it was to be failed (e.g., measurement invariance). This suggests that the pattern of under- reporting in the field may represent widespread hidden invalidity of the measures we use, and therefore pose a threat to many research findings. We highlight the degrees of freedom afforded to researchers in the assessment and reporting of structural validity. Similar to the better-known concept of p-hacking, we introduce the concept of validity hacking (v-hacking) and argue that it should be acknowledged and addressed.

[Version 3 (uploaded 21 April 2020) provides corrected list of co-authors and commenters; the ms. is otherwise unchanged from Versions 1 and 2.] Scientific interest in unintended discrimination that can result from implicit attitudes and stereotypes (implicit biases) has produced a large corpus of empirical findings. In addition to much evidence for validity and usefulness of Implicit Association Test (IAT) measures, there have been psychological critiques of empirical findings and theoretical disagreements about interpretation of IAT findings. Because of public attention drawn by the concept of implicit bias, commercial and other applications based on the concept of implicit bias have been developed by non-psychologists—some of these applications are not appropriately guided by the existing body of research findings. This article is in 5 parts: (1) review of best practices for research use of IAT measures, (2) summary of what has been confidently learned from empirical research using IAT measures, (3) accepted and controversial theoretical interpretations of IAT findings, (4) significant questions about the IAT and implicit bias that still await answer, and (5) questions arising in attempts to apply research findings to remedy unintended discrimination due to implicit biases.

No abstract

Evaluative conditioning (EC) is one of the most widely-studied procedures for establishing and changing attitudes. The surveillance-task (Olson & Fazio, 2001) is a highly cited EC paradigm, and one that is claimed to generate attitudes without awareness. The potential for EC effects to occur without awareness continues to fuel conceptual, theoretical, and applied developments. Yet few published studies have used this task, and most are characterized by small samples and small effect sizes. We conducted a high-powered (N =1478), preregistered close replication of the original surveillance-task study. We obtained evidence for a small EC effect when ‘aware’ participants were excluded using the original criterion – therefore replicating the original effect. However, no such effect emerged when three other awareness criteria were used. We suggest that there is a need for caution when using evidence for the surveillance task effect to make theoretical and practical claims about ‘unaware’ EC effects.