Estimating the Reproducibility of Experimental Philosophy

Abstract

For scientific theories grounded in empirical data, replicability is a core principle, for at least two reasons. First, unless we accept to have scientific theories rest on the authority of a small number of researchers, empirical studies should be replicable, in the sense that its methods and procedure should be detailed enough for someone else to conduct the same study. Second, for empirical results to provide a solid foundation for scientific theorizing, they should also be replicable, in the sense that most attempts at replicating the original study that produced them would yield similar results. The XPhi Replicability Project is primarily concerned with replicability in the second sense, that is: the replicability of results. In the past year, several projects have shed doubt on the replicability of key findings in psychology, and most notably social psychology. Because the methods of experimental philosophy have often been close to the ones used in social psychology, it is only natural to wonder to which extent the results experimental philosophers ground their theory are replicable. The aim of the XPhi Replicability Project is precisely to reach a reliable estimate of the replicability of empirical results in experimental philosophy. To this end, several research teams across the world will replicate around 40 studies in experimental philosophy, some among the most cited, others drawn at random. The results of the project will be published in a special issue of the Review of Philosophy and Psychology dedicated to the topic of replicability in cognitive science.The official website of the project can be found here :https://sites.google.com/site/thexphireplicabilityproject/homeThe project can also be followed on social medias:https://twitter.com/XPhiReplicationhttps://www.facebook.com/XPhiReplicabilityProject/

How Replicable Are Links Between Personality Traits and Consequential Life Outcomes? The Life Outcomes of Personality Replication Project

“…Despite high statistical power, the RPP observed a replication success rate of only 36% (when success was defined as a statistically significant result in the expected direction) and found that the replication effects were, on average, only half as strong as the original effects (Open Science Collaboration, 2015). Similar projects in economics and experimental philosophy have also obtained replicability estimates considerably lower than would be expected in the absence of published false positives, although results have varied somewhat across projects (Camerer et al, 2016;Cova et al, 2018). These findings reinforce concerns about the replicability of behavioral science and suggest that replicability may vary both between and within disciplines.…”

“…These findings reinforce concerns about the replicability of behavioral science and suggest that replicability may vary both between and within disciplines. For example, replicability appears to be higher for original studies that (a) examined main effects rather than interactions, (b) reported intuitive rather than surprising results, and (c) obtained a greater effect size, sample size, and strength of evidence (Camerer et al, 2016;Cova et al, 2018;Open Science Collaboration, 2015).…”

“…The main goal of the LOOPR Project was to estimate the overall replicability of the personality-outcome literature. We therefore adopted an approach that attempted to replicate a large number of original effects from many studies, with one replication attempt per effect and relatively brief outcome measures (Camerer et al, 2016;Cova et al, 2018;Open Science Collaboration, 2015). An alternative approach would be to replicate a smaller number of effects with lengthier measures or multiple replication attempts per effect (i.e., a many-labs approach; Ebersole et al, 2016;Hagger et al, 2016;Klein et al, 2014).…”

Addiction, Identity, Morality

“…It would be concerning, then, if they turned out to be due to some idiosyncratic feature of Jim and his poetic ambitions. To address this issue, and in light of ongoing concerns about replicability in psychology and experimental philosophy (Earp and Trafimow 2015;Cova et al 2018;LeBel et al 2018), we decided to conduct a preregistered replication and extension study, in which we presented participants (between subjects) with four structurally similar vignettes-in addition to the one about Jim-describing other characters with a wide range of personal attributes and goals, but all with the shared feature of experiencing moral improvement versus deterioration as a result of starting or stopping the use of a drug.…”

“…However, it may not be so simple. Recent work in psychology and experimental philosophy (see Knobe 2016;Knobe and Nichols 2008;Knobe et al 2012;Cova et al 2018) suggests that intuitions about changed identity are shaped not just by the fact of some change in demeanor after a transformative event (cf. Paul 2014), but rather by the particular kind of change that occurs.…”