Speaking of success: Real-world experiments for sustainability transformations and causal inference

Abstract

Reports of successful sustainability-oriented real-world experiments can be categorized by the type of explanatory approaches employed, namely, the variance approach, which looks for correlations, and the process approach, which draws causal inferences in a narrative way. Their validity could benefit from a more critical reflection on the data and methods used.This paper examines a sample of 20 sustainability-oriented real-world experiment reports from 2006 to 2020, with the aim of uncovering the logic and methods used to demonstrate their success, that means, to show that they achieved their stated objectives. Following a distinction often made in the social sciences, I look for features of either the variance or the process approach to causal inference. I find that reports of transition experiments, socio-technical experiments, and community-based interventions display characteristic features of the process approach. Reports of trials, pilots, and field experiments, on the other hand, mostly use the variance approach to demonstrate success. An important observation is the limited recognition of possible biases related to the methods used or the data. I describe a number of possible biases that may be of importance in the context of sustainability-oriented real-world experiments. Important examples include measurement errors and biases in participant selection. Recognising the biases and correcting them where necessary can strengthen the validity of the findings obtained and help other researchers in designing their experiments.