2022
DOI: 10.1073/pnas.2203150119
|View full text |Cite|
|
Sign up to set email alerts
|

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Abstract: This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater i… Show more

Help me understand this report
View preprint versions

Search citation statements

Order By: Relevance

Paper Sections

Select...
1

Citation Types

1
73
0

Year Published

2022
2022
2024
2024

Publication Types

Select...
3
3
1

Relationship

0
7

Authors

Journals

citations
Cited by 103 publications
(74 citation statements)
references
References 36 publications
1
73
0
Order By: Relevance
“…Meta research projects of the 'Many analysts, one dataset' type, in which many researchers are testing the same hypothesis with the same dataset, demonstrate that researcher degrees of freedom easily lead to entirely different conclusions. In a recent study relying on international survey data, researchers were asked to estimate the association between immigration and public support for government provision of welfare (Breznau et al, 2022). 25% of estimates were significantly negative, 17% were positive, and 58% had a confidence interval including 0.…”
Section: Researcher Degrees Of Freedom Facilitate Widely Different Co...mentioning
confidence: 99%
“…Meta research projects of the 'Many analysts, one dataset' type, in which many researchers are testing the same hypothesis with the same dataset, demonstrate that researcher degrees of freedom easily lead to entirely different conclusions. In a recent study relying on international survey data, researchers were asked to estimate the association between immigration and public support for government provision of welfare (Breznau et al, 2022). 25% of estimates were significantly negative, 17% were positive, and 58% had a confidence interval including 0.…”
Section: Researcher Degrees Of Freedom Facilitate Widely Different Co...mentioning
confidence: 99%
“… Variation in reported results from Brady and Finnigan ( 4 ) which Breznau et al. ( 1 ) replicate. Point estimates for 3 variables: percent foreign born, net migration, and change in percent foreign born.…”
mentioning
confidence: 93%
“…In an impressive effort, Breznau et al. ( 1 ) (henceforth, BRW) report a many-analyst collaboration where multiple teams were involved in analyzing the same data and hypothesis: that immigration undermines public support for social policy. Like other such studies ( 2 , 3 ), the results show considerable variation and are sure to ignite debate.…”
mentioning
confidence: 99%
See 2 more Smart Citations