Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the “Many Analysts, One Data Set” Project

Abstract

In 2018, Silberzahn, Uhlmann, Nosek, and colleagues published an article in which 29 teams analyzed the same research question with the same data: Are soccer referees more likely to give red cards to players with dark skin tone than light skin tone? The results obtained by the teams differed extensively. Many concluded from this widely noted exercise that the social sciences are not rigorous enough to provide definitive answers. In this article, we investigate why results diverged so much. We argue that the main reason was an unclear research question: Teams differed in their interpretation of the research question and therefore used diverse research designs and model specifications. We show by reanalyzing the data that with a clear research question, a precise definition of the parameter of interest, and theory-guided causal reasoning, results vary only within a narrow range. The broad conclusion of our reanalysis is that social science research needs to be more precise in its “estimands” to become credible.

Keep your enemies close: Adversarial collaborations will improve behavioral science.

“…And many psychological constructs cannot be directly observed, so operationalizations can easily drift away from the original constructs of interest (e.g., Kovera & Evelo, 2021). Different investigators' decisions about how to operationalize variables (Schweinsberg et al, 2021) and analyze data (Silberzahn et al, 2018) can lead to starkly different conclusions (but also see Auspurg & Brüderl, 2021).…”

Moderators of reputation effects in peer-to-peer online markets: a meta-analytic model selection approach

“…Rather, the use of multilevel methodology might be an indication of more sophisticated data collections and better controls for correlated groups. This conjecture could be tested through a re-analyses of studies that used multilevel techniques [5,12,56]. The larger N effect is surprising and can certainly not be seen as a reason for smaller N studies, but is also not completely unknown in the literature [59].…”

“…However, this study cannot dive deep into the specific mechanisms that underlie these moderating effects, because of the limited information available for each individual study. Dedicated studies that consciously look at variations in certain moderators are necessary to investigate in more detail the different causal explanation behind the moderations [5,12,56].…”

“…To reduce the number of missing values, we estimate the mean price in datasets with missing values based on product mean prices from similar datasets. 5 In this way, we obtained for 17 of the 55 datasets an estimated product price. In our analysis, we use the log-transformed average price.…”

“…htm. 5 The estimation is based on similar datasets with reported product mean prices. For instance, datasets with similar types of products (e.g., SD card and U disk), year of data collection (within 5 years), and same usage condition (i.e.…”