Beyond Experiments

Abstract

It is often claimed that only experiments can support strong causal inferences and therefore they should be privileged in the behavioral sciences. We disagree. Overvaluing experiments results in their overuse both by researchers and decision makers and in an underappreciation of their shortcomings. Neglect of other methods often follows. Experiments can suggest whether X causes Y in a specific experimental setting; however, they often fail to elucidate either the mechanisms responsible for an effect or the strength of an effect in everyday natural settings. In this article, we consider two overarching issues. First, experiments have important limitations. We highlight problems with external, construct, statistical-conclusion, and internal validity; replicability; and conceptual issues associated with simple X causes Y thinking. Second, quasi-experimental and nonexperimental methods are absolutely essential. As well as themselves estimating causal effects, these other methods can provide information and understanding that goes beyond that provided by experiments. A research program progresses best when experiments are not treated as privileged but instead are combined with these other methods.

Asymmetrical Effects of Sleep and Emotions in Daily Life

“…Experimental studies have documented similar findings, such that a total night of sleep deprivation or partial sleep deprivation (e.g., 4 h/night) can result in increased negative emotions, such as anxiety, stress, and sadness (Kahn et al, 2013;Meerlo et al, 2008;Palmer & Alfano, 2017). Although informative, cross-sectional and experimental methods are limited in their abilities to elucidate the degree to which sleep can modulate affective experiences that occur naturally in daily life (Newman & Stone, 2019;Diener et al, 2022). Further, the links between sleep, stress, and emotion are likely bidirectional (Walker & Harvey, 2010), a process that cannot be easily or effectively tested by cross-sectional or experimental methods.…”

Spontaneous suppression in dating couples: Social and physiological correlates of suppressing negative and positive emotions during negative and positive conversations

“…First, much of past work has focused on either short-term effects of instructed suppression (e.g., Gross and Levenson, 1993) or trait-level correlates of habitual suppression (e.g., Gross and John, 2003). In other words, much of our understanding of suppression is not based on uninstructed instances of suppression (i.e., spontaneous suppression) and, thus, offers limited insight into how suppression operates when people spontaneously employ it (Diener et al, 2022). Second, little existing research has addressed factors that shape the social and physiological correlates of spontaneous suppression.…”

Effectiveness and equity evaluation of an insurance‐wide telephone‐counseling program for self‐management of chronic diseases: The Health Coach Study

“…, Indeed, it has been argued (Imbens & Rubin, 2015) that, in contrast to the ITT effect that provides an estimate of the average effect of treatment assignment on the outcome, treatment noncompliance models provide unbiased estimates of the average treatment on the treated, a more informative measure of treatment effectiveness 6 . In addition, the need to attain participant's permission to receive either the IG or the CG and the use of incentives to maximize compliance in a randomized clinical trial can lead to limits on the generalization of the results of the trial (Diener et al, in press; see Relton et al, 2010 for a comparison of designs). Cartwright and Hardie (2012) have documented the many failed educational and public‐health interventions based on prior randomized clinical trials that used participants and enhanced interventions that were not representative of the target populations and treatments of interest.…”