Data quality, experimental artifacts, and the reactivity of the psychological subject matter

Abstract

While the term “reactivity” has come to be associated with specific phenomena in the social sciences, having to do with subjects’ awareness of being studied, this paper takes a broader stance on this concept. I argue that reactivity is a ubiquitous feature of the psychological subject matter and that this fact is a precondition of experimental research, while also posing potential problems for the experimenter. The latter are connected to the worry about distorted data and experimental artifacts. But what are experimental artifacts and what is the most productive way of dealing with them? In this paper, I approach these questions by exploring the ways in which experimenters in psychology simultaneously exploit and suppress the reactivity of their subject matter in order to produce experimental data that speak to the question or subject matter at hand. Highlighting the artificiality of experimental data, I raise (and answer) the question of what distinguishes a genuine experimental result from an experimental artifact. My analysis construes experimental results as the outcomes of inferences from the data that take material background assumptions as auxiliary premises. Artifacts occur when one or more of these background assumptions are false, such that the data do not reliably serve the purposes they were generated for. I conclude by laying out the ways in which my analysis of data quality is relevant to, and informed by, recent debates about the replicability of experimental results.

Growth From Uncertainty: Understanding the Replication ‘Crisis’ in Infant Cognition

“…Another question is that raised in section 3.2 regarding the balance between making stimuli that are appropriate for infants and concerns about ecological validity and generalizability. The concern from ecological validity is that the stimuli are so different from life as encountered in the real world that one needs to carefully justify the claim that they are tapping into the same cognitive abilities that babies use “in the wild.” The concern from generalizability is that the stimuli may be testing a very specific cognitive ability (e.g., an infant’s theory of cartoon bears and mice) rather than the indefinitely flexible ability to track goals, which is the real target of investigation (Feest 2022; Packer and Moreno-Dulcey 2013).…”

“…The problems arise when researchers fail to acknowledge their work for what it is: a process of building outward from an uncertain foundation. An important lesson being learned from the replication crisis is that this starting point needs to be made more explicit (Bringmann et al 2022;Feest 2022;Sikorski and Andreoletti 2023).…”

“…Addressing this issue more substantively requires a closer analysis of the fragility of experimental effects, which remains a topic for another article (although seeFeest [2022],Kominsky et al [2022], andvan Bavel [2016] for contributions in this line).…”

Characterizing and Measuring Racial Discrimination in Public Health Research

“…Here I appeal to the idea of reactivity, or the disposition of participants to respond to experimental features. Experimentation requires designing experimental conditions that generate the right kind of reactivity while suppressing other reactivity (Feest 2022). Reactivity is crucial for experimentation because it provides researchers access to psychological processes and their outcomes.…”

The elephant's other legs: What some sciences actually do

“…Or is the problem that the measurements are either not capturing the relevant concept or not measuring the same concept across different studies? Feest (2022), for instance, argues that experimental psychologists must address three distinct reactivity challengesall related to the ways psychological subject matter have dispositions to react to experimental contextsin designing their experiments; Almaatouq et al's call for integrative experiment design across a design space suggests another challenge to designing experiments where genuine results can be distinguished from artifacts. Thus, our suggestion is that, for commensurability to obtain in the design space as the authors advocate, researchers must first precisely specify the operative concepts.…”