The individual-level precision of implicit measures

Abstract

Implicit attitude measures are widely used across many fields of psychological science. One core goal of these measures is to provide precise information which can be diagnostic of an individual person’s attitude. To date, little progress has been made towards this goal. We argue that this is because psychologists have not yet even quantified individual-level precision in these tasks, much less been able to calibrate their measures towards it. We use bootstrapping to fit confidence intervals to individual-level implicit measure scores using a large dataset (N = ~23,413 individuals) of six different implicit measures assessing three different attitude domains (race, politics, and self-esteem). Our analyses focused on the ability of these measures to distinguish participants from neutral attitudes and each other, while also evaluating the width and coverage of confidence intervals. Despite some variation, all measures exhibited substantial room for improvement. We recommend that researchers in future should use metrics of individual-level precision to calibrate their tasks, both in the context of implicit measures and with tasks in psychological science more broadly.

On the measurement of relational responding

“…The human observations are derived from published and unpublished data. Specifically, RAI data are taken from the time 1 assessments in Experiment 2 of Cummins et al (2023) and data from unpublished studies using the 8-relation RAI. These data were only used if the RAI was the first measure participants completed and if participants were typically developed adults between 18 and 40 years old.…”

“…Although metrics for individual-level precision of scores in such tasks are not currently used often within CBS (or in psychology more generally), such metrics do exist. For example, statistical approaches such as the standard error of measurement (Dudek, 1979), the rescaled estimates of true scores (RETS; Schmukle, 2023), and bootstrapping (Mooney et al, 1993) can all be used to quantify the (im)precision of a measurement procedure at the individual-level; see for example Hussey's (2020) application of bootstrapping in the Implicit Relational Assessment Procedure, and Cummins and Hussey (2023).…”