Investigating Variation in Replicability: A “Many Labs” Replication Project

Klein, Richard Anthony; Ratliff, Kate A.; Vianello, Michelangelo; Adams, Reginald B.; Bahník, Štěpán; Bernstein, Michael H.; Bocian, Konrad; Brandt, Mark; Brooks, Beach Smith; Brumbaugh, Claudia Chloe; Cemalcılar, Zeynep; Chandler, Jesse; Cheong, Winnee; Davis, William E.; Devos, Thierry; Eisner, Matthew; Frankowska, Natalia; Furrow, David; Galliani, Elisa Maria; Hasselman, Fred; Hicks, Joshua A.; Hovermale, James F.; Hunt, S. Jane; Huntsinger, Jeffrey R.; IJzerman, Hans; John, Melissa-Sue; Joy-Gaba, Jennifer A.; Kappes, Heather Barry; Krueger, Lacy E.; Kurtz, Jaime L.; Levitan, Carmel; Mallett, Robyn K.; Morris, Wendy L.; Nelson, Anthony J.; Nier, Jason A.; Packard, Grant; Pilati, Ronaldo; Rutchick, Abraham M.; Schmidt, Kathleen; Skorinko, Jeanine; Smith, Robert C.; Steiner, Troy G.; Storbeck, Justin; Swol, Lyn M. Van; Thompson, Donna; Veer, Anna Elisabeth van 't; Vaughn, Leigh Ann; Vranka, Marek; Wichman, Aaron L.; Woodzicka, Julie A.; Nosek, Brian A.

doi:10.31234/osf.io/ux8ef

Cited by 85 publications

(141 citation statements)

References 24 publications

Supporting

Mentioning

131

Contrasting

Unclassified

Order By: Relevance

“…Concerns have been raised about the credibility of published results following John Ioannidis' landmark essay, "Why most published findings are false" 3 , and the identification of a considerable number of studies that turned out to be false positives 4,5 . In response, several large-scale replication projects were initiated in the fields of psychology, experimental economics, and the social sciences more generally [6][7][8][9][10][11][12][13][14] to systematically evaluate a large sample of findings through direct replication.…”

Section: Introductionmentioning

confidence: 99%

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

et al. 2021

View full text Add to dashboard Cite

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package “pooledmaRket” and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.

show abstract

Section: Introductionmentioning

confidence: 99%

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In the Israeli sample, 34 out of 504 individuals (7%) recommended their responses be excluded. This is below the average exclusion rate for online studies (Klein et al, 2014). Those who recommended not to use their responses did not differ significantly in age, Welch’s t-test (37.73) = 2.81, p = .10, or gender, χ 2 = 2.28, p = .320, from the rest of the sample.…”

Section: Methodsmentioning

confidence: 71%

Making sense of the past to understand the present: Attributions for historical trauma predict contemporary social and political attitudes

Hirschberger

Kahn

Hanke

2021

Group Processes & Intergroup Relations

View full text Add to dashboard Cite

Research indicates that the memory of collective trauma influences attitudes towards contemporary social and political issues. We suggest that the specific attributions for trauma that members of victim and perpetrator groups make provide a more nuanced understanding of this relationship. Thus, we constructed and validated a measure of attributions for the Holocaust. Then, we ran a preregistered study on representative samples in Germany ( N = 504) and Israel ( N = 469) to examine whether attributing the Holocaust to essentialist or contextual causes influences attitudes towards the immigration crisis and the Israeli–Palestinian conflict. Results indicated that, among Germans, attributing the Holocaust to German character was associated with positive attitudes to immigration via collective guilt. Among Israelis, attributions to German character were associated with negative attitudes to non-Jewish immigration, a hawkish stance in the Palestinian–Israeli conflict, and pro-Israel attitudes via a sense of perpetual victimization. Results reveal how attributions about past trauma affect contemporary social and political attitudes among victims and perpetrators.

show abstract

“…Suppose that we are interested in CROSS-SECTIONAL INTEGRATIVE DATA ANALYSIS modeling the effect of extraversion on conscientiousness. The first and simplest fixed-effects approach is a regression model using the pooled data (e.g., Klein et al, 2014),…”

Section: Aggregated Regressionmentioning

confidence: 99%

Modeling approaches for cross-sectional integrative data analysis: Evaluations and recommendations.

Wilcox

Wang

2023

Psychological Methods

View full text Add to dashboard Cite

Integrative data analysis (IDA) jointly models participant-level data from multiple studies. In psychology, IDA has been conducted using different fixed-effects and multilevel modeling (MLM) approaches. However, evaluations regarding the performance of these models in an IDA context are limited. The goal of this article is to evaluate three fixed-effects models (aggregated vs. disaggregated vs. study-specific coefficients regressions) and two MLMs (fixed-slope vs. random-slopes MLM) for cross-sectional IDA. Using a simulation study with study sample sizes and numbers of studies consistent with applied IDA (e.g., two to 35 studies), we evaluated estimation bias and type I error rates for participant-level and studylevel effects and variance components for these models; for the MLMs, we evaluated different estimation methods (i.e., constrained vs. unconstrained variance estimation and five degrees of freedom methods). Disaggregated and study-specific coefficients regressions and both MLMs yielded fixed effects estimates with ignorable bias, but only the random-slopes MLM fully modeled between-study heterogeneity and, consequently, provided well-controlled type I error rates for testing both fixed effects. Overall, we found that MLMs could be feasible under IDA conditions with three to six studies and well-chosen estimation methods. A real-data IDA example is used to illustrate and compare the application of the five models. We hope our results will help researchers choose appropriate modeling methods when conducting IDA.

show abstract

Investigating Variation in Replicability: A “Many Labs” Replication Project

Cited by 85 publications

References 24 publications

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

Predicting replicability—Analysis of survey and prediction market data from large-scale forecasting projects

Making sense of the past to understand the present: Attributions for historical trauma predict contemporary social and political attitudes

Modeling approaches for cross-sectional integrative data analysis: Evaluations and recommendations.

Contact Info

Product

Resources

About