Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Schweinsberg, Martin; Feldman, Michael D.; Staub, Nicola; Akker, Olmo R. van den; Aert, Robbie C. M. van; Assen, Marcel A.L.M. van; Liu, Yang; Althoff, Tim; Heer, Jeffrey; Kale, Alex; Mohamed, Zainab; Amireh, Hashem; Prasad, Vaishali Venkatesh; Bernstein, Abraham; Robinson, Emily V.; Snellman, Kaisa; Sommer, S. Amy; Otner, Sarah M. G.; Robinson, David; Madan, Nikhil; Silberzahn, Raphael; Goldstein, Pavel; Tierney, Warren; Murase, Toshio; Mandl, Benjamin; Viganola, Domenico; Strobl, Carolin; Schaumans, Catherine; Kelchtermans, Stijn; Naseeb, Chan; Garrison, S. Mason; Yarkoni, Tal; Chan, Christian S.; Adie, Prestone; Alaburda, Paulius; Albers, Casper J.; Alspaugh, Sara; Alstott, Jeff; Nelson, Andrew A.; Rubia, Eduardo Ariño de la; Adbi, Arzi; Bahník, Štěpán; Baik, Jason Min; Balling, Laura Winther; Banker, Sachin; Baranger, David A.A.; Barr, Dale J.; Barros-Rivera, Brenda A.; Bauer, Matt; Enuh, Blaise Manga; Boelen, Lisa; Carbonell, Katerina Bohle; Briers, Robert A.; Burkhard, Oliver; Canela, Miguel Ángel; Castrillo, Laura; Catlett, Timothy; Chen, Olivia; Clark, Michael; Cohn, Brent; Coppock, Alex; Cugueró-Escofet, Natàlia; Curran, Paul; Cyrus-Lai, Wilson; Dai, David; Riva, Giulio Valentino Dalla; Danielsson, Henrik; Russo, Rosaria; Silva, Niko de; Derungs, Curdin; Dondelinger, Frank; Souza, Carolina Duarte de; Dube, Blessing; Dubova, Marina; Dunn, Ben Mark; Edelsbrunner, Peter A.; Finley, Sara; Fox, Nick C.; Gnambs, Timo; Gong, Yuanyuan; Grand, Erin; Greenawalt, Brandon; Dan, Han; Hanel, Paul H. P.; Hong, Antony B.; Hood, David D.; Hsueh, Justin; Huang, Lilian; Hui, Kent; Hultman, Keith A.; Javaid, Azka; Jiang, Lily; Jong, Jonathan; Kamdar, Jash; Kane, David; Kappler, Gregor; Kaszubowski, Erikson; Kavanagh, Christopher; Khabsa, Madian; Kleinberg, Bennett; Kouros, Jens; Krause, Heather; Krypotos, Angelos Miltiadis; Lavbič, Dejan; Lee, Rui Ling; Leffel, Timothy; Lim, Wei Yang; Liverani, Silvia; Loh, Bianca; Lønsmann, Dorte; Low, Jia Wei; Lu, Alton; MacDonald, Kyle; Madan, Christopher R.; Madsen, Lasse Hjorth; Maimone, Christina; Mangold, Alexandra; Marshall, Adrienne M.; Matskewich, Helena Ester; Mavon, Kimia; McLain, Katherine L.; McNamara, Amelia; McNeill, Mhairi; Mertens, Ulf K.; Miller, David I.; Moore, Ben; Moore, Andrew; Nantz, Eric; Nasrullah, Ziauddin; Nejković, Valentina; Nell, Colleen S.; Nilsonne, Gustav; Nolan, Rory; O'Brien, Christopher E.; O’Neill, Patrick; O’Shea, Kieran J.; Olita, Toto; Otterbacher, Jahna; Palsetia, Diana; Pereira, Bianca; Pozdniakov, Ivan; Protzko, John; Reyt, Jean Nicolas; Riddle, Travis; Ali, Amal Ridhwan Omar; Ropovik, Ivan; Rosenberg, Joshua M.; Rothen, Stéphane; Schulte‐Mecklenbeck, Michael; Sharma, Nirek; Shotwell, Gordon; Skarzynski, Martin; Stedden, William; Stodden, Victoria; Stoffel, Martin; Stoltzman, Scott; Subbaiah, Subashini; Tatman, Rachael; Thibodeau, Paul H.; Tomkins, Sabina; Valdivia, Ana; Woestijne, Gerrieke B. Druijff-van de; Viana, Laura; Villeséche, Florence; Wadsworth, W. Duncan; Wanders, Florian; Watts, Krista; Wells, Jason D.; Whelpley, Christopher E.; Won, Andy; Wu, Lawrence; Yip, Arthur; Youngflesh, Casey; Yu, Ju Chi; Zandian, Arash; Zhang, Leilei; Zibman, Chava; Uhlmann, Eric Luis

doi:10.1016/j.obhdp.2021.02.003

Cited by 84 publications

(57 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The idea of inviting different analysis teams to answer the same research question using the same data is relatively novel (Silberzahn and Uhlmann, 2015;see Aczel et al, 2021 for general guidelines); we are aware of three papers in neuroscience (Botvinik-Nezer et al, 2020;Fillard et al, 2011;Maier-Hein et al, 2017), one in microeconomics (Huntington-Klein et al, 2021), and eight in psychology, three of which pertain to cognitive modeling (Boehm et al, 2018;Dutilh et al, 2019;Starns et al, 2019) while the remaining five are from other fields of psychology (Bastiaansen et al, 2020;Salganik et al, 2020;Schweinsberg et al, 2021;Silberzahn et al, 2018;van Dongen et al, 2019). Most similar to the current work are the projects that applied a many-analysts approach to perform statistical inference on the relation between two variables, such as skin color and red cards in soccer (Silberzahn et al, 2018), scientist gender and verbosity (Schweinsberg et al, 2021), or amygdala activity and stress (van Dongen et al, 2019). While the exact focus of previous many-analysts projects varied (e.g., experience sampling, fMRI preprocessing, predictive modeling, proof of the many-analysts concept), the take-home messages were rather consistent: all papers showed that different yet equally justifiable analytic choices result in very different outcomes, sometimes with statistically significant effects in opposite directions (e.g., Schweinsberg et al, 2021;Silberzahn et al, 2018).…”

Section: A Many-analysts Approachmentioning

confidence: 99%

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

Hoogeveen¹,

Sarafoglou²

2022

Preprint

View full text Add to dashboard Cite

The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset (N=10,535 participants from 24 countries). We recruited 120 analysis teams to investigate (1) whether religious people self-report higher well-being, and (2) whether the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion (i.e., whether it is considered normal and desirable to be religious in a given country). In a two-stage procedure, the teams first created an analysis plan and then executed their planned analysis on the data. For the first research question, all but 3 teams reported positive effect sizes with credible/confidence intervals excluding zero (median reported beta = 0.120). For the second research question, this was the case for 65% of the teams (median reported beta = 0.039). While most teams applied (multilevel) linear regression models, there was considerable variability in the choice of items used to construct the independent variables, the dependent variable, and the included covariates.

show abstract

Section: A Many-analysts Approachmentioning

confidence: 99%

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

Hoogeveen¹,

Sarafoglou²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It is important to note that whenever the co-authors’ behavior is the subject of the study then they should be regarded similarly to human participants respecting ethical and data protection regulations. Useful templates for project advertisement and analyst surveys can be found in Silberzahn et al, 2018 ; Schweinsberg et al, 2021 .…”

Section: Multi-analyst Guidancementioning

confidence: 99%

“…As an extension, the co-analysts can be asked to record considered but rejected analysis choices and the reasoning behind their choices (e.g., by commented code, log-books, or dedicated solutions such as DataExplained [ Schweinsberg et al, 2021 ]). These logs can reflect where and why co-analysts diverge in their choices.…”

Section: Multi-analyst Guidancementioning

confidence: 99%

“…They overlaid each teams’ confidence in their findings and added additional information about analytical paths in adjacent columns ( Supplementary file 1 , Table 1 ). For a project with a relatively small number of effect sizes for continuous outcomes, Schweinsberg et al used interval plots combined with an indication of analytical choices underlying each estimate ( Schweinsberg et al, 2021 ; Figure 3). Olsson Collentine et al used funnel plots (Figure 2 in Olsson-Collentine et al, 2020 ), and Patel et al used volcano plots to depict numerous, diverse outcomes with an intuitive depiction of clustering, akin to a multiverse analysis (Figures 1 and 2 in Patel et al, 2015 ).…”

Section: Multi-analyst Guidancementioning

confidence: 99%

See 1 more Smart Citation

Consensus-based guidance for conducting and reporting multi-analyst studies

Aczél

Szászi

Nilsonne

et al. 2021

eLife

Self Cite

View full text Add to dashboard Cite

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

show abstract

“…The problem is more relevant in social sciences, where the decisions of the modeller have a substantial effect on the results because of the higher indetermination in the definition of the prediction task (e.g., [23]). Likewise, when complex databases are used, different decisions may lead to opposite conclusions [24].…”

mentioning

confidence: 99%

Validation of Machine Learning Models for Structural Dam Behaviour Interpretation and Prediction

Mata

Salazar

Barateiro

et al. 2021

Water

View full text Add to dashboard Cite

The main aim of structural safety control is the multiple assessments of the expected dam behaviour based on models and the measurements and parameters that characterise the dam’s response and condition. In recent years, there is an increase in the use of data-based models for the analysis and interpretation of the structural behaviour of dams. Multiple Linear Regression is the conventional, widely used approach in dam engineering, although interesting results have been published based on machine learning algorithms such as artificial neural networks, support vector machines, random forest, and boosted regression trees. However, these models need to be carefully developed and properly assessed before their application in practice. This is even more relevant when an increase in users of machine learning models is expected. For this reason, this paper presents extensive work regarding the verification and validation of data-based models for the analysis and interpretation of observed dam’s behaviour. This is presented by means of the development of several machine learning models to interpret horizontal displacements in an arch dam in operation. Several validation techniques are applied, including historical data validation, sensitivity analysis, and predictive validation. The results are discussed and conclusions are drawn regarding the practical application of data-based models.

show abstract

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Cited by 84 publications

References 78 publications

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

A Many-Analysts Approach to the Relation Between Religiosity and Well-being

Consensus-based guidance for conducting and reporting multi-analyst studies

Validation of Machine Learning Models for Structural Dam Behaviour Interpretation and Prediction

Contact Info

Product

Resources

About