Reporting in Experimental Philosophy: Current Standards and Recommendations for Future Practice

Abstract

Recent replication crises in psychology and other fields have led to intense reflection about the validity of common research practices. Much of this reflection has focussed on reporting standards, and how they may be related to the questionable research practices that could underlie a high proportion of irreproducible findings in the published record. As a developing field, it is particularly important for Experimental Philosophy to avoid some of the pitfalls that have beset other disciplines. To this end, here we provide a detailed, comprehensive assessment of current reporting practices in Experimental Philosophy. We focus on the quality of statistical reporting and the disclosure of information about study methodology. We assess all the articles using quantitative methods (n = 134) that were published over the years 2013-2016 in 29 leading philosophy journals. We find that null hypothesis significance testing is the prevalent statistical practice in Experimental Philosophy, although relying solely on this approach has been criticised in the psychological literature. To augment this approach, various additional measures have become commonplace in other fields, but we find that Experimental Philosophy has adopted these only partially: 53% of the papers report an effect size, 28% confidence intervals, 1% examined prospective statistical power and 5% report observed statistical power. Importantly, we find no direct relation between an article's reporting quality and its impact (numbers of citations). We conclude with recommendations for authors, reviewers and editors in Experimental Philosophy, to facilitate making research statistically-transparent and reproducible.Rev.Phil.Psych.

Critical Analysis of the Research on Digital Literacy

“…The relevance of research is determined by its contribution to existing knowledge. Improving reporting, design, and research behavior is critical in maintaining the credibility and overall dependability of findings (Komala & Firdaus, 2023) (Bottesini et al, 2023) (Polonioli et al, 2018). To evaluate the validity of the data extraction form, researchers conducted a pre-test to ensure its clarity and efficacy in retrieving pertinent information from thirteen studies.…”

Hasty Generalizations Are Pervasive in Experimental Philosophy: A Systematic Analysis

“…Because scientific databases (e.g., Scopus) do not monitor all relevant philosophy journals, we adopted an approach by Polonioli et al (2021), who combined the quantitative ranking provided by the h-index with established informal polls, such as the Leiter Report journal ranking, to form a list of twenty journals frequently publishing x-phi. From these journals, we focused on those that Polonioli et al found to have published four or more x-phi articles over three years.…”

Preregistration of studies with existing data

“…Currently, preregistration is variably common the different scientific disciplines. Whereas a preregistration is mandatory in many medical journals to publish the results of clinical trials, other fields are only taking their first steps in applying preregistration for their studies (e.g., experimental philosophy; see Polonioli et al 2018). Therefore, the elements that should be included in the preregistration still differ widely across scientific fields and, for some types of research designs and in certain scientific fields, no generally accepted standards or templates for preregistration are available as of now (e.g., Haven et al 2020).…”

“…Although the idea of preregistering studies has been around since the 1960's, researchers have only relatively recently started using preregistration for research in different areas (e.g., psychology, philosophy, social sciences). As such, for these fields, preregistration is a relatively new tool and it is still being further developed and evaluated (e.g., Haven et al 2020;Polonioli et al 2018). Given the relatively quick developments in the field of open science in psychology and beyond, we anticipate that a study's preregistration will become more common for different types of research designs and scientific disciplines.…”