Artificiality, Reactivity, and Demand Effects in Experimental Economics

Abstract

A series of recent debates in experimental economics have associated demand effects with the artificiality of the experimental setting and have linked it to the problem of external validity. In this paper, we argue that these associations can be misleading, partly because of the ambiguity with which “artificiality” has been defined, but also because demand effects and external validity are related in complex ways. We argue that artificiality (understood as unfamiliarity of the experimental environment) may be directly as well as inversely correlated with demand effects. We also distinguish between the demand effects of experimentation and the reactions that they may trigger and that might endanger experimental validity. We conclude that economists should pay more attention to the way in which subjects construe the experimental task and learn to exploit subjects’ reactivity to expectations in their experiments

Reactivity in social scientific experiments: what is it and how is it different (and worse) than a Placebo effect?

“…3 and 4 above). It should be noted that apprehension to evaluation is only one of the many potential triggers of reactivity, where other common, well-known manifestations or mechanisms include the subjects' reactions to the perceived authority of the experimenter, the participant's zeal for being "a good subject" (or the opposite uncooperative desire to "boycott" an experiment), or the pervasive and understandable participants' active search for cues and second guesses about what the experiment is really about (Jimenez-Buedo and Guala 2016). By conceptualizing the phenomenon of reactivity in this way, we can better see what distinguishes reactivity in an experimental context from the more encompassing, general phenomenon of reactivity in observational research.…”

“…In this regard, Zizzo's more ambitious conceptual project is careful: in his framework, experimenter demand effects are not in themselves a problem but have the potential to create one whenever experimental subjects can correctly guess the objectives of the experiment, yet, as Jimenez-Buedo and Guala (2016) have argued, this approach neglects that often many economic experiments successfully align the incentives of subjects and experimenters through monetary rewards that are meant, precisely, to inform experimental subjects what exactly is sought of them, or in other words, what the objective of the experiment really is. Thus, and although Zizzo's identification of this condition seems to fit the DG case nicely, it does not constitute the best grounds for a general elucidation of these conditions.…”

“…In this way, and according to critics, players in the DG are merely trying to perform the role of "good subjects" by adjusting their behavior to expectations, or, more specifically, adjusting to what they think it is expected of them as subjects (Bardsley 2008;Zizzo 2010). Alternatively, others have argued that relevant inferences from the DG and other similarly abstract games are still possible: both in the lab and in the field, subjects' behavior depends on other people's expectations and thus the DG provides a useful setting to study how subjects choose to adhere to the normative cues that the experimental setting provides (Levitt andList 2007, Jimenez-Buedo andGuala 2016).…”

Enlarged Empirical Economics and the Quest for Validity: Facing the Ontological Intricacies of the Social Domain

“…Buedo and Guala (2016) try to disentangle the skein of artificiality by distinguishing three meanings of the term (subjects' awareness of being studied, unfamiliarity with the experimental tasks, and gap between the experimental setting and the target situation), which leads them to conclude that the relationship between artificiality, demand effects, and external validity is far from being univocal.…”

Data quality, experimental artifacts, and the reactivity of the psychological subject matter

“…1 Similar issues have been addressed under the label of a "demand" effect, i.e., the effects of experimental subjects (consciously or unconsciously) trying to figure out what is being asked of them in the experiment and acting accordingly (e.g., Orne, 1962). Writing in the philosophy of economics, Jimenez-Buedo and Guala (2016) choose the term "reactivity" to describe the phenomenon underlying such effects: "[W]e shall call reactivity the phenomenon that occurs when individuals alter their behaviour because of the awareness of being studied" (Jimenez-Buedo & Guala, 2016, 11; see also Jimenez-Buedo, 2021). Now, it would seem that the mere fact of research subjects altering their behavior when studied is not necessarily a problem, unless the altered behavior somehow gives rise to faulty data and/or distorted experimental results.…”