Research practices and statistical reporting quality in 250 economic psychology master's theses: a meta-research investigation

Abstract

The replicability of research findings has recently been disputed across multiple scientific disciplines. In constructive reaction, the research culture in psychology is facing fundamental changes, but investigations of research practices that led to these improvements have almost exclusively focused on academic researchers. By contrast, we investigated the statistical reporting quality and selected indicators of questionable research practices (QRPs) in psychology students' master's theses. In a total of 250 theses, we investigated utilization and magnitude of standardized effect sizes, along with statistical power, the consistency and completeness of reported results, and possible indications of p-hacking and further testing. Effect sizes were reported for 36% of focal tests (median r = 0.19), and only a single formal power analysis was reported for sample size determination (median observed power 1 − β = 0.67). Statcheck revealed inconsistent p-values in 18% of cases, while 2% led to decision errors. There were no clear indications of p-hacking or further testing. We discuss our findings in the light of promoting open science standards in teaching and student supervision.

Misprediction of affective outcomes due to different evaluation modes: Replication and extension of two distinction bias experiments by Hsee and Zhang (2004)

“…We intended to assess the replication results by comparing effect sizes between the original and replication studies (see , but the original results in the paper by were not reported in sufficient detail to allow for such comparisons, which further points to the importance of complete statistical reporting in scientific articles (see Bakker & Wicherts, 2011;Olsen et al, 2019). Nonetheless, (in Supplemental Materials) we present effect size comparisons between the replication and original studies, where possible.…”

Quality assessment of scientific manuscripts in peer review and education

“…Furthermore, theses differ from published manuscripts in many aspects that may be relevant for their review and evaluation. For example, previous research showed that theses report effect sizes more frequently than published manuscripts (Augusteijn et al, 2022;Fritz et al, 2013;Krishna & Peter, 2018), on average use larger samples (Olsen, et al, 2019), and conduct a priori power analysis more often (Krishna & Peter, 2018;Kühberger et al, 2014). These differences could also indicate that students and researchers consider different aspects in assessing the quality of these manuscripts.…”

Evaluating the Pedagogical Effectiveness of Study Preregistration in the Undergraduate Dissertation

“…For example, the Framework of Open and Reproducible Research Training (FORRT; www.forrt.org) includes preregistration as one of the six pillars of effective reproducibility training, including at the undergraduate level. Others have suggested that “most study programmes should offer easy ways of implementing preregistration in empirical research seminars” (Olson et al, 2019) because of the potential for preregistration to promote “critical reflections of research practices” and improve students’ statistics literacy (Olson et al, 2019). As Pownall (2020b) also argued, the process of embedding preregistration of undergraduate dissertations largely complements current practices in dissertation supervision.…”