Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Abstract

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

Appraising Contemporary Social Work Research: Meta-Research on Statistical Reporting, Statistical Power, and Evidential Value

“…The purpose of this meta-research study is to assess statistical practices in contemporary social work research. It parallels recent work in psychology (Nuijten, 2018), neuroscience (Szucs & Ioannidis, 2017), medicine (de Vries, Roest, Turner, & de Jonge, 2018; Herrera-Perez et al, 2019), ecology (Fraser, Parker, Nakagawa, Barnett, & Fidler, 2018;Kelly, 2006), and other scientific disciplines (see Ioannidis, Fanelli, Dunne, & Goodman, 2015); while introducing a formal program of inquiry to the social work profession. The study is comprised of three complementary parts, focusing on the 1) statistical reporting practices, 2) statistical power, and the 3) evidential value of a subsection of research studies.…”

“…Others specific recommendations include, 1) replacement of frequentist statistics with Bayesian statistics (discussed in some detail in the next chapter) (Dienes & Mclatchie, 2018), 2) pre-registration of study protocols and 3) encouragement of confirmatory (rather than exploratory) analyses (Wagenmakers et al, 2012), 4) sharing of data, code/syntax, codebooks, and other materials in public repositories (Lindsay, 2018), 5) large-scale, multi-site collaborations (to among other things, increase statistical power in rare or hard to reach populations) (e.g. Moshontz et al, 2018), 6) the encouragement of independent replication studies (McNeely & Warner, 2015;Nosek & Errington, 2019), 7) meta-research (Ioannidis, Fanelli, Dunne, & Goodman, 2015;Nuijten, 2018), and 8) educational reform (e.g. Dunleavy & Lacasse, 2020).…”

“…Meta-research is a broad and burgeoning area of study, which concerns the methods, reporting, and reproducibility of research (Ioannidis et al, 2015). Simply put meta-research is "research on research" (Nuijten, 2018). Meta-research can focus on any aspect of the research process; from study planning/design through dissemination/postpublication, as well as the context in which such research is produced (e.g.…”