Meta-analysis to integrate effect sizes within an article: Possible misuse and Type I error inflation.

Ueno, Taiji; Fastrich, Greta M.; Murayama, Kou

doi:10.1037/xge0000159

Cited by 53 publications

(44 citation statements)

References 65 publications

(134 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As would be the case with any random sampling process, there are some exceptions, but the bulk of the published evidence clearly supports the main claim (aligning with the rationale behind a p-curve analysis; Simonsohn et al, 2014). Although we summarize the key results of our prior experiments across several publications here, we only make qualitative comparisons between these results, given recent demonstrations that internal meta-analyses can problematically overstate the strength of evidence for an effect (Ueno et al, 2016;Vosgerau et al, 2018).…”

Section: Overview Of Resultsmentioning

confidence: 99%

Comparative inspiration: From puzzles with pigeons to novel discoveries with humans in risky choice

Madan¹,

Ludvig²

2018

Preprint

View full text Add to dashboard Cite

Both humans and non-human animals regularly encounter decisions involving risk and uncertainty. This paper provides an overview of our research program examining risky decisions in which the odds and outcomes are learned through experience in people and pigeons. We summarize the results of 15 experiments across 8 publications, with a total of over 1300 participants. We highlight 4 key findings from this research: (1) people choose differently when the odds and outcomes are learned through experience compared to when they are described; (2) when making decisions from experience, people overweight values at or near the ends of the distribution of experienced values (i.e., the best and the worst, termed the “extreme-outcome rule”), which leads to more risk seeking for relative gains than for relative losses; (3) people show biases in self-reported memory whereby they are more likely to report an extreme outcome than an equally-often experienced non-extreme outcome, and they judge these extreme outcomes as having occurred more often; and (4) under certain circumstances pigeons show similar patterns of risky choice as humans, but the underlying processes may not be identical. This line of research has stimulated other research in the field of judgement and decision making, illustrating how investigations from a comparative perspective can lead in surprising directions.

show abstract

Section: Overview Of Resultsmentioning

confidence: 99%

Comparative inspiration: From puzzles with pigeons to novel discoveries with humans in risky choice

Madan¹,

Ludvig²

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…The sampling plan should also preclude the researcher to conduct a particular study multiple times, and only present the “best” study (i.e., the one with the most desirable results). The use of multiple small studies instead of a larger one is an effective (yet problematic) strategy to find at least one statistically significant result (Bakker et al, 2012) and small underpowered studies can also be pooled by means of a meta-analysis in an ad hoc manner to obtain a statistically significant result (Ueno et al, 2016). Hence, we call the following researcher DF, D7: Failing to specify the sampling plan and allowing for running (multiple) small studies.…”

Section: Design Phasementioning

confidence: 99%

Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

et al. 2016

View full text Add to dashboard Cite

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size estimates. In this review article, we present an extensive list of 34 degrees of freedom that researchers have in formulating hypotheses, and in designing, running, analyzing, and reporting of psychological research. The list can be used in research methods education, and as a checklist to assess the quality of preregistrations and to determine the potential for bias due to (arbitrary) choices in unregistered studies.

show abstract

“…The sampling plan should also preclude the researcher to conduct a particular study multiple times, and only present the "best" study (i.e., the one with the most desirable results). The use of multiple small studies instead of a larger one is an effective (yet problematic) strategy to find at least one statistically significant result and small underpowered studies can also be pooled by means of a meta-analysis in an ad hoc manner to obtain a statistically significant result (Ueno, Fastrich, & Murayama, 2016). Hence, we call the following researcher DF, D7: Failing to specify the sampling plan and allowing for running (multiple) small studies.…”

Section: Power and Sampling Planmentioning

confidence: 99%

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Veldkamp¹

2017

Preprint

View full text Add to dashboard Cite

THE HUMAN FALLIBILITY OF SCIENTISTSDealing with error and bias in academic researchRecent studies have highlighted that not all published findings in the scientific literature are trustworthy, suggesting that currently implemented control mechanisms such as high standards for the reporting of research methods and results, peer review, and replication, are not sufficient. In psychology in particular, solutions are sought to deal with poor reproducibility and replicability of research results. In this dissertation project I considered these problems from the perspective that the scien¬tific enterprise must better recognize the human fallibility of scientists, and I examined potential solutions aimed at dealing with human error and bias in psychological science. First, I studied whether the human fallibility of scientists is actually recognized (Chapter 2). I examined the degree to which scientists and lay people believe in the storybook image of the scientist: the image that scientists are more objective, rational, open-minded, intelligent, honest and communal than other human beings. The results suggested that belief in this storybook image is strong, particularly among scientists themselves. In addition, I found indications that scientists believe that scientists like themselves fit the storybook image better than other scientists. I consider scientist’s lack of acknowledgement of their own fallibility problematic, because I believe that critical self-reflection is the first line of defense against potential human error aggravated by confirmation bias, hindsight bias, motivated reasoning, and other human cognitive biases that could affect any professional in their work. Then I zoomed in on psychological science and focused on human error in the use of null the most widely used statistical framework in psychology: hypothesis significance testing (NHST). In Chapters 3 and 4, I examined the prevalence of errors in the reporting of statistical results in published articles, and evaluated a potential best practice to reduce such errors: the so called ‘co-pilot model of statistical analysis’. This model entails a simple code of conduct prescribing that statistical analyses are always conducted independently by at least two persons (typically co-authors). Using statcheck, a software package that is able to quickly retrieve and check statistical results in large sets of published articles, I replicated the alarmingly high error rates found in earlier studies. Although I did not find support for the effectiveness of the co-pilot model in reducing these errors, I proposed several ways to deal with human error in (psychological) research and suggested how the effectiveness of the proposed practices might be studied in future research. Finally, I turned to the risk of bias in psychological science. Psychological data can often be analyzed in many different ways. The often arbitrary choices that researchers face in analyzing their data are called researcher degrees of freedom. Researchers might be tempted to use these researcher degrees of freedom in an opportunistic manner in their pursuit of statistical significance (often called p-hacking). This is problematic because it renders research results unreliable. In Chapter 5 I presented a list of researcher degrees of freedom in psychological studies, focusing on the use of NHST. This list can be used to assess the potential for bias in psychological studies, it can be used in research methods education, and it can be used to examine the effectiveness of a potential solution to restrict oppor¬tunistic use of RDFs: study pre-registration. Pre-registration requires researchers to stipulate in advance the research hypothesis, data collection plan, data analyses, and what will be reported in the paper. Different forms of pre-registration are currently emerging in psychology, mainly varying in terms of the level of detail with respect to the research plan they require researchers to provide. In Chapter 6, I assessed the extent to which current pre-registrations restricted opportunistic use of the researcher degrees of freedom on the list presented in Chapter 5. We found that most pre-registrations were not sufficiently restrictive, but that those that were written following better guidelines and requirements restricted opportunistic use of researcher degrees of freedom considerably better than basic pre-registrations that were written following a limited set of guidelines and requirements. We concluded that better instructions, specific questions, and stricter requirements are necessary in order for pre-registrations to do what they are supposed to do: to protect researchers from their own biases.

show abstract

Meta-analysis to integrate effect sizes within an article: Possible misuse and Type I error inflation.

Cited by 53 publications

References 65 publications

Comparative inspiration: From puzzles with pigeons to novel discoveries with humans in risky choice

Comparative inspiration: From puzzles with pigeons to novel discoveries with humans in risky choice

Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Contact Info

Product

Resources

About