Increasing Transparency Through a Multiverse Analysis

Steegen, Sara; Tuerlinckx, Francis; Gelman, Andrew; Vanpaemel, Wolf

doi:10.1177/1745691616658637

Cited by 1,095 publications

(1,191 citation statements)

References 43 publications

Supporting

Mentioning

1,176

Contrasting

Unclassified

Order By: Relevance

“…The use of NHST involves many, often arbitrary decisions that its users need to make Bakker & Wicherts, 2014a;John et al, 2012;Kriegeskorte, Simmons, Bellgowan, & Baker, 2009;Nieuwenhuis, Forstmann, & Wagenmakers, 2011;Simmons et al, 2011;Simonsohn et al, 2014a;Steegen, Tuerlinckx, Gelman, & Vanpaemel, 2016;van Aert et al, 2016;. Making as many of those decisions before the analyses are conducted or even before the study takes place forces authors to think through all potential pitfalls in the design and the analyses and to collaboratively formulate an analysis plan.…”

Section: Discussionmentioning

confidence: 99%

“…Such potential for bias is particularly severe for experiments that study subtle effects with relatively small samples. Hence, we need an appropriate way to deal with researcher DFs; one way to assess the relevance of choices is to report all potentially relevant analyses either as a traditional sensitivity analyses or as a multiverse analysis (Steegen et al, 2016). Another solution is that the data are available for independent reanalysis after publication, although this is not always possible due to low sharing rates .…”

Section: Discussionmentioning

confidence: 99%

“…Most of the researcher DFs have been described in previous publications Bakker & Wicherts, 2014b;Chambers, 2013;Francis, 2013;John et al, 2012;Kriegeskorte et al, 2009;Nieuwenhuis et al, 2011;Simmons et al, 2011;Simonsohn et al, 2014a;Steegen et al, 2016;van Aert et al, 2016;Wagenmakers et al, 2011). The researcher DFs on our list are invariably inspired by actual research we have encountered as researchers, replicators, re-analyzers, and critical readers of published and unpublished works.…”

Section: Chapter 5 Abstractmentioning

confidence: 97%

“…The latter use was implemented in the current study, where we constructed a coding protocol based directly on this list. As explicated in Chapter 5 , the list itself was created in a qualitative manner based on both the existing literature Bakker & Wicherts, 2014b;Chambers, 2013;Francis, 2013;John et al, 2012;Kriegeskorte et al, 2009;Nieuwenhuis et al, 2011;Simmons et al, 2011;Simonsohn et al, 2014a;Steegen et al, 2016;van Aert et al, 2016;Wagenmakers et al, 2011) and additional considerations.…”

Section: Researcher Degrees Of Freedom Assessedmentioning

confidence: 99%

See 3 more Smart Citations

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Veldkamp¹

2017

Preprint

View full text Add to dashboard Cite

THE HUMAN FALLIBILITY OF SCIENTISTSDealing with error and bias in academic researchRecent studies have highlighted that not all published findings in the scientific literature are trustworthy, suggesting that currently implemented control mechanisms such as high standards for the reporting of research methods and results, peer review, and replication, are not sufficient. In psychology in particular, solutions are sought to deal with poor reproducibility and replicability of research results. In this dissertation project I considered these problems from the perspective that the scien¬tific enterprise must better recognize the human fallibility of scientists, and I examined potential solutions aimed at dealing with human error and bias in psychological science. First, I studied whether the human fallibility of scientists is actually recognized (Chapter 2). I examined the degree to which scientists and lay people believe in the storybook image of the scientist: the image that scientists are more objective, rational, open-minded, intelligent, honest and communal than other human beings. The results suggested that belief in this storybook image is strong, particularly among scientists themselves. In addition, I found indications that scientists believe that scientists like themselves fit the storybook image better than other scientists. I consider scientist’s lack of acknowledgement of their own fallibility problematic, because I believe that critical self-reflection is the first line of defense against potential human error aggravated by confirmation bias, hindsight bias, motivated reasoning, and other human cognitive biases that could affect any professional in their work. Then I zoomed in on psychological science and focused on human error in the use of null the most widely used statistical framework in psychology: hypothesis significance testing (NHST). In Chapters 3 and 4, I examined the prevalence of errors in the reporting of statistical results in published articles, and evaluated a potential best practice to reduce such errors: the so called ‘co-pilot model of statistical analysis’. This model entails a simple code of conduct prescribing that statistical analyses are always conducted independently by at least two persons (typically co-authors). Using statcheck, a software package that is able to quickly retrieve and check statistical results in large sets of published articles, I replicated the alarmingly high error rates found in earlier studies. Although I did not find support for the effectiveness of the co-pilot model in reducing these errors, I proposed several ways to deal with human error in (psychological) research and suggested how the effectiveness of the proposed practices might be studied in future research. Finally, I turned to the risk of bias in psychological science. Psychological data can often be analyzed in many different ways. The often arbitrary choices that researchers face in analyzing their data are called researcher degrees of freedom. Researchers might be tempted to use these researcher degrees of freedom in an opportunistic manner in their pursuit of statistical significance (often called p-hacking). This is problematic because it renders research results unreliable. In Chapter 5 I presented a list of researcher degrees of freedom in psychological studies, focusing on the use of NHST. This list can be used to assess the potential for bias in psychological studies, it can be used in research methods education, and it can be used to examine the effectiveness of a potential solution to restrict oppor¬tunistic use of RDFs: study pre-registration. Pre-registration requires researchers to stipulate in advance the research hypothesis, data collection plan, data analyses, and what will be reported in the paper. Different forms of pre-registration are currently emerging in psychology, mainly varying in terms of the level of detail with respect to the research plan they require researchers to provide. In Chapter 6, I assessed the extent to which current pre-registrations restricted opportunistic use of the researcher degrees of freedom on the list presented in Chapter 5. We found that most pre-registrations were not sufficiently restrictive, but that those that were written following better guidelines and requirements restricted opportunistic use of researcher degrees of freedom considerably better than basic pre-registrations that were written following a limited set of guidelines and requirements. We concluded that better instructions, specific questions, and stricter requirements are necessary in order for pre-registrations to do what they are supposed to do: to protect researchers from their own biases.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Chapter 5 Abstractmentioning

confidence: 97%

Section: Researcher Degrees Of Freedom Assessedmentioning

confidence: 99%

See 2 more Smart Citations

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Veldkamp¹

2017

Preprint

View full text Add to dashboard Cite

show abstract

“…As a result, an outlier analysis is very likely to produce a multiverse of outcomes (cf. Steegen, Tuerlinckx, Gelman, & Vanpaemel, 2016).…”

mentioning

confidence: 99%

A Practical Illustration of Methods to Deal with Potential Outliers: A Multiverse Outlier Analysis of Study 3 from Brummelman, Thomaes, Orobio de Castro, Overbeek, and Bushman ()

Verkoeijen

Polak

Bouwmeester

2018

Collabra: Psychology

View full text Add to dashboard Cite

Recently, Brummelman, Thomaes, Orobio de Castro, Overbeek, and Bushman (2014: Study 3) demonstrated that inflated praise benefits challenge seeking of children with high self-esteem, but harms challenge seeking of children with low self-esteem. In the present paper, we examined the original data set on model-fit and prediction outliers according to various reasonable criteria and norms. Subsequently, we carried out a multiverse outlier re-analysis on the data of Brummelman and colleagues' Study 3, employing the same analytical approach as the original authors did but excluding outliers. Out of the twelve re-analyses in the multiverse, six demonstrated that removing only a small number of outliers rendered the originally reported crucial interaction effect between self-esteem and type of praise non-significant and produced a sizeable reduction of the effect size. The present paper illustrates the use of reporting outlier analyses, which lies in allowing a critical evaluation of the empirical evidence and offering a more complete picture that enhances future studies in the field.

show abstract

Social relations and presence of others predict bystander intervention: Evidence from violent incidents captured on CCTV

Liebst

Philpot

Bernasco

et al. 2019

Aggressive Behavior

View full text Add to dashboard Cite

Are individuals willing to intervene in public violence? Half a century of research on the “bystander effect” suggests that the more bystanders present at an emergency, the less likely each of them is to provide help. However, recent meta‐analytical evidence questions whether this effect generalizes to violent emergencies. Besides the number of bystanders present, an alternative line of research suggests that pre‐existing social relations between bystanders and conflict participants are important for explaining whether bystanders provide help. The current paper offers a rare comparison of both factors—social relations and the number of bystanders present—as predictors of bystander intervention in real‐life violent emergencies. We systematically observed the behavior of 764 bystanders across 81 violent incidents recorded by surveillance cameras in Copenhagen, Denmark. Bystanders were sampled with a case–control design, their behavior was observed and coded, and the probability of intervention was estimated with multilevel regression analyses. The results confirm our predicted association between social relations and intervention. However, rather than the expected reversed bystander effect, we found a classical bystander effect, as bystanders were less likely to intervene with increasing bystander presence. The effect of social relations on intervention was larger in magnitude than the effect of the number of bystanders. We assess these findings in light of recent discussions about the influence of group size and social relations in human helping. Further, we discuss the utility of video data for the assessment of real‐life bystander behavior.

show abstract

Increasing Transparency Through a Multiverse Analysis

Cited by 1,095 publications

References 43 publications

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

Doctoral thesis: The human fallibility of scientists - dealing with error and bias in academic research

A Practical Illustration of Methods to Deal with Potential Outliers: A Multiverse Outlier Analysis of Study 3 from Brummelman, Thomaes, Orobio de Castro, Overbeek, and Bushman ()

Social relations and presence of others predict bystander intervention: Evidence from violent incidents captured on CCTV

Contact Info

Product

Resources

About