Many analysts, one dataset: Making transparent how variations in analytical choices affect results

Abstract

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in analysis of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.

Open Science: What, Why, and How

“…Open science is a term for some of the proposed reforms to make scientific practices more transparent and to increase the availability of information that allows others to evaluate and use the research (Nosek et al, 2015). We acknowledge that many, or even most, of the proposed reforms of the open science movement are not new (see Bastian, 2016;Spellman, 2015, Table A1).…”

“…Journals choose which, if any, component at which level to adopt and can tailor each component to their preferences or needs (Nosek et al, 2015). As of September 2017, over 3,200 journals, including the Science and Nature family of journals, and about 65 organizations have become signatories of the TOP Guidelines.…”

“…Second, hindsight bias ("I knew it all along") helps researchers to transform evidence that might have countered a theory into evidence that ultimately supports the researcher's (perhaps new) position. Combine these two biases with flexibility in design and analysis (John et al, 2012) and mixed messages on which statistical techniques are really best (Silberzahn et al, 2015), and the conditions are right for getting the wrong answer. These biases pervade daily life, and scientists are unfortunately not immune to them (Chambers, 2017).…”

“…Additionally, researchers may benefit from sharing study data and materials by receiving increased citations (Belter, 2014;Piwowar & Vision, 2013). Scientists should create stronger norms for citing shared data and materials and-crucially-highly value such citations (Belter, 2014;Chen, Downs, & Shumacher, 2013;Nosek et al, 2015).…”

Promoting Open Science to Increase the Trustworthiness of Evidence in Special Education

“…This includes reproducibility of methods to demonstrate that application of the original analytic code to the original data achieved the reported result. It also includes estimating the robustness of evidence by evaluating the extent to which findings using the same data are consistent across a variety of reasonable analytic decisions, such as how to treat outliers and which covariates to include (LeBel et al, in press;Silberzahn et al, 2017).…”