Blind Analysis as a Correction for Confirmatory Bias in Physics and in Psychology

MacCoun, Robert J.; Perlmutter, S.

doi:10.1002/9781119095910.ch15

Cited by 28 publications

(29 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Foremost, we propose that complex datasets should be analyzed using multiple analysis pipelines, preferably by more than one researcher, who would be blinded to the hypotheses of interest 23 and to the results obtained using other pipelines. Achieving such "multiverse analysis" 24 at scale will require the development of fully automated configurable statistical analysis tools (e.g.…”

Section: Discussionmentioning

confidence: 99%

Variability in the analysis of a single neuroimaging dataset by many teams

Botvinik‐Nezer

Holzmeister

Camerer

et al. 2019

Preprint

178

209

View full text Add to dashboard Cite

SummaryData analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.

show abstract

Section: Discussionmentioning

confidence: 99%

Variability in the analysis of a single neuroimaging dataset by many teams

Botvinik‐Nezer

Holzmeister

Camerer

et al. 2019

Preprint

178

209

View full text Add to dashboard Cite

show abstract

“…Thus, pre-data collection review can be especially helpful to researchers trying to avoid confirmation bias. Finally, blinding (MacCoun & Perlmutter, 2017) is a process in which data are perturbed or condition labels are scrambled in order to decouple data exploration from knowledge of a study's results. Using a holdout sample (in which exploration occurs on half of the data, while the remaining half is "held out" for later confirmatory testing) is another technique with a similar purpose.…”

Section: Insights From Psychological Science Itselfmentioning

confidence: 99%

Open Science: What, Why, and How

Spellman¹,

Gilbert²,

Corker³

2017

Preprint

View full text Add to dashboard Cite

Open science is a collection of actions designed to make scientific processes more transparent and results more accessible. Its goal is to build a more replicable and robust science; it does so using new technologies, altering incentives, and changing attitudes. The current movement toward open science was spurred, in part, by a recent series of unfortunate events within psychology and other sciences. These events include the large number of studies that have failed to replicate and the prevalence of common research and publication procedures that could explain why. Many journals and funding agencies now encourage, require, or reward some open science practices, including pre-registration, providing full materials, posting data, distinguishing between exploratory and confirmatory analyses, and running replication studies. Individuals can practice and promote open science in their many roles as researchers, authors, reviewers, editors, teachers, and members of hiring, tenure, promotion, and awards committees. A plethora of resources are available to help scientists, and science, achieve these goals. Keywords: data sharing, file drawer problem, open access, open science, preregistration, questionable research practices, replication crisis, reproducibility, scientific integrityThanks to Brent Donnellan (big thanks!), Daniël Lakens, Calvin Lai, Courtney Soderberg, and Simine Vazire OPEN SCIENCE 3 Open Science When we (the authors) look back a couple of years, to the earliest outline of this chapter, the open science movement within psychology seemed to be in its infancy. Plenty of people were pointing to problems in psychology research, collecting archival data to support the claims, and suggesting how science could be improved. Now it seems that the open science movement has reached adolescence. Things are happening-and they are happening quickly. New professional organizations are being formed to uncover and facilitate ways to improve science, often through new technology, and some old organizations are adopting new procedures to remedy problems created by past practices, often involving revising journal policies. Researchers are changing the way they teach, practice, and convey science. And scientific information (and opinions) are traveling fast. In blogs, tweets, Facebook groups, op eds, science journalism, circulation of preprints, post-print comments, video talks, and so on, more people are engaged in communicating science, and hoping to improve science, than ever before. Thus, any new technology, new procedure, new website, or new controversy we describe is likely to be superseded (or solved) even by the time this chapter is published. But the core values of open science should remain. The "Open Science" MovementScience is about evidence: observing, measuring, collecting, and analyzing evidence. And it is about evidence that can be shared across observers and, typically, although not necessarily exactly (Merton, 1973;Popper, 1959), replicated later. Science is about testing hypotheses, using inductiv...

show abstract

“…The example shows how blinding can prevent a real and substantial effect from being downgraded from a confirmatory to an exploratory finding. 5 A comprehensive assessment of the strengths and weaknesses of the different blinding methods demands a series of Monte Carlo simulations similar to those presented in MacCoun and Perlmutter (2017). In such simulations one can vary effect size, sample size, direction of effect size, and particular aspects of the design or the data.…”

Section: Blinding As Integral Part Of Preregistrationmentioning

confidence: 99%

Flexible yet fair: blinding analyses in experimental psychology

2019

View full text Add to dashboard Cite

The replicability of findings in experimental psychology can be improved by distinguishing sharply between hypothesis-generating research and hypothesis-testing research. This distinction can be achieved by preregistration, a method that has recently attracted widespread attention. Although preregistration is fair in the sense that it inoculates researchers against hindsight bias and confirmation bias, preregistration does not allow researchers to analyze the data flexibly without the analysis being demoted to exploratory. To alleviate this concern we discuss how researchers may conduct blinded analyses (MacCoun and Perlmutter in Nature 526:187-189, 2015). As with preregistration, blinded analyses break the feedback loop between the analysis plan and analysis outcome, thereby preventing cherry-picking and significance seeking. However, blinded analyses retain the flexibility to account for unexpected peculiarities in the data. We discuss different methods of blinding, offer recommendations for blinding of popular experimental designs, and introduce the design for an online blinding protocol.

show abstract

Blind Analysis as a Correction for Confirmatory Bias in Physics and in Psychology

Cited by 28 publications

References 70 publications

Variability in the analysis of a single neuroimaging dataset by many teams

Variability in the analysis of a single neuroimaging dataset by many teams

Open Science: What, Why, and How

Flexible yet fair: blinding analyses in experimental psychology

Contact Info

Product

Resources

About