Underreporting in Psychology Experiments

Franco, Annie; Malhotra, Neil; Simonovits, Gábor

doi:10.1177/1948550615598377

Cited by 115 publications

(114 citation statements)

References 22 publications

Supporting

Mentioning

106

Contrasting

Order By: Relevance

“…This is a practice that a substantial proportion of psychologists admitted to (Agnoli et al, 2017;John et al, 2012; see also http://psychdisclosure.org). This practice has also been observed directly, when comparing published and unpublished results in a known population of conducted studies (Chan, Hrobjartsson, Haahr, Gotzsche, & Altman, 2004;Franco et al, 2016;O'Boyle, Banks, & Gonzalez-Mule, 2017).…”

Section: Solutionsmentioning

confidence: 58%

“…This becomes even more problematic, if only the analyses that "worked" are reported and presented as if they were planned from the start (Kerr, 1998;Wagenmakers, Wetzels, Borsboom, Maas, & Kievit, 2012). Survey results show that many psychologists admit to such "questionable research practices" (QRPs; Agnoli, Wicherts, Veldkamp, Albiero, & Cubelli, 2017;John et al, 2012), and use of study registers and later disclosures by researchers provide direct evidence that indeed some of these practices are quite common (Franco, Malhotra, & Simonovits, 2016;LeBel et al, 2013).…”

Section: Bias and Errorsmentioning

confidence: 99%

See 1 more Smart Citation

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

show abstract

Section: Solutionsmentioning

confidence: 58%

Section: Bias and Errorsmentioning

confidence: 99%

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…Among the research practices studied by John et al, this practice of not reporting all dependent measures showed quite high prevalence estimates (John et al, 2012). Additionally, direct evidence indicates that psychological researchers often choose among different outcome measures (Franco et al, 2016;LeBel, Borsboom, Giner-Sorolla, Hasselman, Peters, Ratliff, & Smith, 2013). In the medical literature on randomized clinical trials, this researcher DF is often called outcome switching and the bias it introduces is called outcome reporting bias (Chan, Hrobjartsson, Haahr, Gotzsche, & Altman, 2004;Kirkham et al, 2010;Weston et al, 2016).…”

Section: Dependent Variable(s)mentioning

confidence: 99%

“…Dropping of experimental conditions has been found to be quite common in a survey among psychological researchers (John et al, 2012) and in a study that considered psychological studies from a register (Franco, Malhotra, & Simonovits, 2016). Specifically, a researcher can discard a factor in a multifactorial experiment by pooling the data over the levels of that factor, or the researcher can select certain levels of a discarded factor.…”

Section: Design Phasementioning

confidence: 99%

“…This use may result in statistically significant findings after all, (denoted p-hacking) and may also result in overestimated effect sizes and dissemination of false positive results. In psychology, the opportunistic use of the so-called 'Researcher Degrees of Freedom' (Simmons et al, 2011; constitutes a large problem because its occurrence is estimated to be high Fanelli, 2009;Fiedler & Schwarz, 2015;Franco et al, 2016;John et al, 2012;LeBel, Borsboom, Giner-Sorolla, Hasselman, Peters, Ratliff, & Tucker Smith, 2013;O'Boyle, Banks, & Gonzalez-Mulé, 2014) and its effects are particularly strong for underpowered (small sample) studies that are very common in psychology Cohen, 1962;Maxwell, 2004).…”

Section: Chapter 6 Abstractmentioning

confidence: 99%

See 1 more Smart Citation

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Veldkamp¹

2017

Preprint

View full text Add to dashboard Cite

THE HUMAN FALLIBILITY OF SCIENTISTSDealing with error and bias in academic researchRecent studies have highlighted that not all published findings in the scientific literature are trustworthy, suggesting that currently implemented control mechanisms such as high standards for the reporting of research methods and results, peer review, and replication, are not sufficient. In psychology in particular, solutions are sought to deal with poor reproducibility and replicability of research results. In this dissertation project I considered these problems from the perspective that the scien¬tific enterprise must better recognize the human fallibility of scientists, and I examined potential solutions aimed at dealing with human error and bias in psychological science. First, I studied whether the human fallibility of scientists is actually recognized (Chapter 2). I examined the degree to which scientists and lay people believe in the storybook image of the scientist: the image that scientists are more objective, rational, open-minded, intelligent, honest and communal than other human beings. The results suggested that belief in this storybook image is strong, particularly among scientists themselves. In addition, I found indications that scientists believe that scientists like themselves fit the storybook image better than other scientists. I consider scientist’s lack of acknowledgement of their own fallibility problematic, because I believe that critical self-reflection is the first line of defense against potential human error aggravated by confirmation bias, hindsight bias, motivated reasoning, and other human cognitive biases that could affect any professional in their work. Then I zoomed in on psychological science and focused on human error in the use of null the most widely used statistical framework in psychology: hypothesis significance testing (NHST). In Chapters 3 and 4, I examined the prevalence of errors in the reporting of statistical results in published articles, and evaluated a potential best practice to reduce such errors: the so called ‘co-pilot model of statistical analysis’. This model entails a simple code of conduct prescribing that statistical analyses are always conducted independently by at least two persons (typically co-authors). Using statcheck, a software package that is able to quickly retrieve and check statistical results in large sets of published articles, I replicated the alarmingly high error rates found in earlier studies. Although I did not find support for the effectiveness of the co-pilot model in reducing these errors, I proposed several ways to deal with human error in (psychological) research and suggested how the effectiveness of the proposed practices might be studied in future research. Finally, I turned to the risk of bias in psychological science. Psychological data can often be analyzed in many different ways. The often arbitrary choices that researchers face in analyzing their data are called researcher degrees of freedom. Researchers might be tempted to use these researcher degrees of freedom in an opportunistic manner in their pursuit of statistical significance (often called p-hacking). This is problematic because it renders research results unreliable. In Chapter 5 I presented a list of researcher degrees of freedom in psychological studies, focusing on the use of NHST. This list can be used to assess the potential for bias in psychological studies, it can be used in research methods education, and it can be used to examine the effectiveness of a potential solution to restrict oppor¬tunistic use of RDFs: study pre-registration. Pre-registration requires researchers to stipulate in advance the research hypothesis, data collection plan, data analyses, and what will be reported in the paper. Different forms of pre-registration are currently emerging in psychology, mainly varying in terms of the level of detail with respect to the research plan they require researchers to provide. In Chapter 6, I assessed the extent to which current pre-registrations restricted opportunistic use of the researcher degrees of freedom on the list presented in Chapter 5. We found that most pre-registrations were not sufficiently restrictive, but that those that were written following better guidelines and requirements restricted opportunistic use of researcher degrees of freedom considerably better than basic pre-registrations that were written following a limited set of guidelines and requirements. We concluded that better instructions, specific questions, and stricter requirements are necessary in order for pre-registrations to do what they are supposed to do: to protect researchers from their own biases.

show abstract

Testing for funnel plot asymmetry of standardized mean differences

Pustejovsky

Rodgers

2019

Research Synthesis Methods

220

150

View full text Add to dashboard Cite

Publication bias and other forms of outcome reporting bias are critical threats to the validity of findings from research syntheses. A variety of methods have been proposed for detecting selective outcome reporting in a collection of effect size estimates, including several methods based on assessment of asymmetry of funnel plots, such as the Egger's regression test, the rank correlation test, and the Trim‐and‐Fill test. Previous research has demonstated that the Egger's regression test is miscalibrated when applied to log‐odds ratio effect size estimates, because of artifactual correlation between the effect size estimate and its standard error. This study examines similar problems that occur in meta‐analyses of the standardized mean difference, a ubiquitous effect size measure in educational and psychological research. In a simulation study of standardized mean difference effect sizes, we assess the Type I error rates of conventional tests of funnel plot asymmetry, as well as the likelihood ratio test from a three‐parameter selection model. Results demonstrate that the conventional tests have inflated Type I error due to the correlation between the effect size estimate and its standard error, while tests based on either a simple modification to the conventional standard error formula or a variance‐stabilizing transformation both maintain close‐to‐nominal Type I error.

show abstract

Underreporting in Psychology Experiments

Cited by 115 publications

References 22 publications

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Testing for funnel plot asymmetry of standardized mean differences

Contact Info

Product

Resources

About