Statistical Reporting Inconsistencies in Experimental Philosophy

Abstract

Statistical reporting inconsistencies in experimental philosophyColombo, Matteo; Duev, Georgi; Nuijten, M.B.; Sprenger, Jan General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.-Users may download and print one copy of any publication from the public portal for the purpose of private study or research -You may not further distribute the material or use it for any profit-making activity or commercial gain -You may freely distribute the URL identifying the publication in the public portal Take down policyIf you believe that this document breaches copyright, please contact us providing details, and we will remove access to the work immediately and investigate your claim. AbstractExperimental philosophy (x-phi) is a young field of research in the intersection of philosophy and psychology. It aims to make progress on philosophical questions by using experimental methods traditionally associated with the psychological and behavioral sciences, such as null hypothesis significance testing (NHST). Motivated by recent discussions about a methodological crisis in the behavioral sciences, questions have been raised about the methodological standards of x-phi. Here, we focus on one aspect of this question, namely the rate of inconsistencies in statistical reporting. Previous research has examined the extent to which published articles in psychology and other behavioral sciences present statistical inconsistencies in reporting the results of NHST. In this study, we used the R package statcheck to detect statistical inconsistencies in x-phi, and compared rates of inconsistencies in psychology and philosophy. We found that rates of inconsistencies in x-phi are lower than in the psychological and behavioral sciences. From the point of view of statistical reporting consistency, x-phi seems to do no worse, and perhaps even better, than psychological science. IntroductionExperimental philosophy (x-phi) is a young field at the intersection of philosophy and psychology that aims to make progress on philosophical questions by using experimental methods traditionally associated with the psychological and behavioral sciences [1][2]. Those sciences are, however, undergoing a methodological crisis regarding the reproducibility and statistical correctness of experimental research [3][4][5][6];. This raises the question of whether x-phi is equally affected by this crisis, or whether notable differences can be found.One aspect of the methodological crisis in psychology is the high rate of reporting inconsistencies in the statistical analysis of data; roughly half of the published papers in psychology contain at least one inconsistent result where the reported p-value does not match the reported value and degrees of freedom of the test statistic. Around one in eight papers contain at least one gross inconsi...

Précis of Philosophy Within Its Proper Bounds

“…Chapter 5 (“Eight Defenses of the Method of Cases”) rebuts eight responses in defense of the method of cases. I defend the experimental competencies of experimental philosophers, in line with recent results suggesting that experimental philosophers’ findings replicate well (Cova et al., forthcoming) and do not suffer from various biases (Colombo, Duev, Nuijten, and Sprenger, ; Stuart, Colaço, & Machery, ms). I also provide evidence that reflection does not influence the judgments made in response to philosophical cases (see also Colaço, Kneer, Alexander, & Machery, ms).…”

Estimating the Reproducibility of Experimental Philosophy

“…4), with few p-values in the .025-.05 range. Finally, recent research by Colombo et al (2017) found that the rate of statistical reporting inconsistencies was lower in experimental philosophy than in others parts of behavioral science. In any case, Liao (2015) does seem to have been right with his prediction, and we cannot exclude the possibility that the higher observed replicability of x-phi findings compared to psychology findings might reflect particular cultural values and research practices within the field.…”

Les pratiques de recherche ouvertes en psychologie

“…Outre le simple aspect rebutant logistiquement parlant ( Campbell et al, 2002 ), ainsi que des barrières légales et éthiques invoquées par les auteurs lorsque l’on demande d’expliquer leur refus de partager leurs données ( Hardwicke & Ioannidis, 2018 ), on peut suspecter qu’une des raisons moins avouables qui pourrait expliquer cette appréhension de rendre accessibles ses données est la crainte qu’une analyse indépendante pointe des erreurs invalidant les conclusions de l’étude. Dans les publications scientifiques, les erreurs statistiques sont plus communes que ce que l’on peut imaginer de prime abord, et ce, dans différentes disciplines, que ce soit en psychologie ( Bakker & Wicherts, 2011 ; Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2016 ; Rouder, Haaf, & Snyder, 2019 ), en médecine ( García-Berthou & Alcaraz, 2004 ; Murphy, 2004 ), en psychiatrie ( Berle & Starcevic, 2007 ), en management ( Bergh, Sharp, Aguinis, & Li, 2017 ) ou en philosophie expérimentale ( Colombo, Duev, Nuijten, & Sprenger, 2018 ). Les erreurs étant humaines (et source d’embarras, Bishop, 2018 ), il serait extraordinairement étonnant de n’en observer aucune.…”