The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices

Giofrè, David; Cumming, Geoff; Fresc, Luca; Boedker, Ingrid; Tressoldi, Patrizio

doi:10.1371/journal.pone.0175583

Cited by 95 publications

(130 citation statements)

References 34 publications

Supporting

Mentioning

118

Contrasting

Unclassified

Order By: Relevance

“…Previous research investigating the effectiveness of the badges in more detail found that after the introduction of the badges, data was more often available, correct, usable, and complete (Kidwell et al, 2016). These results are in line with the finding that journal submission guidelines in general can inspire desirable change in authors' behavior (Giofrè, Cumming, Fresc, Boedker, & Tressoldi, 2017; but see also Morris & Fritz, 2017). Note, however, that our design is observational, which does not allow us to draw a causal conclusion.…”

Section: Effectiveness Open Data Policysupporting

confidence: 82%

Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology

Nuijten

Borghuis

Veldkamp

et al. 2017

Collabra: Psychology

View full text Add to dashboard Cite

In this paper, we present three retrospective observational studies that investigate the relation between data sharing and statistical reporting inconsistencies. Previous research found that reluctance to share data was related to a higher prevalence of statistical errors, often in the direction of statistical significance (Wicherts, Bakker, & Molenaar, 2011). We therefore hypothesized that journal policies about data sharing and data sharing itself would reduce these inconsistencies. In Study 1, we compared the prevalence of reporting inconsistencies in two similar journals on decision making with different data sharing policies. In Study 2, we compared reporting inconsistencies in psychology articles published in PLOS journals (with a data sharing policy) and Frontiers in Psychology (without a stipulated data sharing policy). In Study 3, we looked at papers published in the journal Psychological Science to check whether papers with or without an Open Practice Badge differed in the prevalence of reporting errors. Overall, we found no relationship between data sharing and reporting inconsistencies. We did find that journal policies on data sharing seem extremely effective in promoting data sharing. We argue that open data is essential in improving the quality of psychological science, and we discuss ways to detect and reduce reporting inconsistencies in the literature.

show abstract

Section: Effectiveness Open Data Policysupporting

confidence: 82%

Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology

Nuijten

Borghuis

Veldkamp

et al. 2017

Collabra: Psychology

View full text Add to dashboard Cite

show abstract

Section: Effectiveness Open Data Policysupporting

confidence: 78%

“…(Giofrè et al, 2017;Kidwell et al, 2016). Even though these results seem promising, they should be interpreted with care.…”

Section: Discussionmentioning

confidence: 90%

“…We did find, however, that data sharing policies are strongly related to actual data sharing, which might mean that incentivizing data sharing might be effective (see also Giofrè et al, 2017;Harper & Kim, 2017;Kidwell et al, 2016). This is potentially great news, because raw data are notoriously hard to obtain in psychology via personal communications (Vanpaemel et al, 2015;Vines et al, 2014;Wicherts et al, 2006).…”

Section: Chaptermentioning

confidence: 90%

See 1 more Smart Citation

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

show abstract

“…Trofimovich and Ellis (2015) adopted the Open Science Badges for Language Learning, and several other journals now also value open materials and data in this way (e.g., "Author guidelines for contributors," 2017; "Instructions for contributors," 2017). Kidwell et al (2016) and Giofrè, Cumming, Fresc, Boedker, and Tressoldi (2017) have provided quantitative evidence for the effectiveness of this initiative for the transparency of materials and data. Indeed, partly as a result of these initiatives and push from journal editors, IRIS now holds 24 sets of L2 data, in addition to approximately 3,600 files of materials and analysis protocols.…”

Section: Recommendationmentioning

confidence: 99%

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

et al. 2018

View full text Add to dashboard Cite

Despite its critical role for the development of the field, little is known about replication in second language (L2) research. To better understand replication practice, we first provide a narrative review of challenges related to replication, drawing on recent developments in psychology. This discussion frames and motivates a systematic review, building on syntheses of replication in psychology, education, and L2 research. We coded 67 self-labeled L2 replication studies found across 26 journals for 136 characteristics. We estimated a mean rate of 1 published replication study for every 400 articles, with a mean of 6.64 years between initial and replication studies and a mean of 117 citations of the initial study before a replication was published. Replication studies had an annual mean of 7.3 citations, much higher than averages in linguistics and education. Overlap in authorship between initial and replication studies and the availability of the initial materials both increased the likelihood of a replication supporting the initial findings. Our sample contained no direct (exact) replication attempts, and changes made to initial studies were numerous and wide ranging, which likely obscured, if not undermined, the interpretability of replication studies. To improve the amount and quality of L2 replication research, we propose 16 recommendations relating to rationale, nomenclature, design, infrastructure, and incentivization for collaboration and publication.

show abstract

The influence of journal submission guidelines on authors' reporting of statistics and use of open research practices

Cited by 95 publications

References 34 publications

Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology

Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field

Contact Info

Product

Resources

About