The perils of small stimulus sets: How stimulus selection affects generalizability

Abstract

Psychologists use experiments to understand causal relationships. The effects that we observe are typically shown in subsets of people using specific stimuli, yet we assume these specific effects can generalize to many different populations and circumstances than specifically tested. Here, we provide a clear demonstration of how the set of stimuli chosen in an experiment can dramatically influence the observed results. We asked whether there exists a memory bias for high-calorie food images, as others have previously reported such a bias. Using a 650% increase in the number of stimuli relative to previous reports, we find no evidence of such a bias, and instead show the perils of testing for categorical differences in psychological phenomena when only using a few stimuli in each category. We discuss how this relates to the field more broadly and ways to reduce the influence of stimulus selection on performance.