Statisticians, Roll Up Your Sleeves! There's A Crisis to be Solved

Seibold, Heidi; Charlton, Alethea; Boulesteix, Anne‐Laure; Hoffmann, Sabine

doi:10.1111/1740-9713.01554

Cited by 8 publications

(6 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…is by no means a caricature, but a relatively accurate portrait of how studies in evolutionary biology are at present conducted and reported. The vast majority of investigations in our field still follow the traditional NHST scheme, despite ample exposure of its problems for about a century (Amrhein et al, 2017;Berkson, 1938;Boring, 1919;Gigerenzer, 2018;Greenland, 2017;Hurlbert & Lombardi, 2009;Johnson, 1999;Mayo, 2018;McShane et al, 2019;Oakes, 1986;Rozeboom, 1960;Szucs & Ioannidis, 2017;Yates, 1951;Ziliak & McCloskey, 2008), and despite broad agreement within the community of statisticians that the current state of NHST usage is damaging to science (Amrhein et al, 2019a;Benjamin et al, 2018;Seibold et al, 2021;Wasserstein & Lazar, 2016;Wasserstein et al, 2019).…”

Section: How Widely Are the Above Miscon Cep Ti On S Recog Nized In O...mentioning

confidence: 99%

Why and how we should join the shift from significance testing to estimation

Berner

Amrhein

2022

J of Evolutionary Biology

View full text Add to dashboard Cite

A paradigm shift away from null hypothesis significance testing seems in progress. Based on simulations, we illustrate some of the underlying motivations. First, p‐values vary strongly from study to study, hence dichotomous inference using significance thresholds is usually unjustified. Second, ‘statistically significant’ results have overestimated effect sizes, a bias declining with increasing statistical power. Third, ‘statistically non‐significant’ results have underestimated effect sizes, and this bias gets stronger with higher statistical power. Fourth, the tested statistical hypotheses usually lack biological justification and are often uninformative. Despite these problems, a screen of 48 papers from the 2020 volume of the Journal of Evolutionary Biology exemplifies that significance testing is still used almost universally in evolutionary biology. All screened studies tested default null hypotheses of zero effect with the default significance threshold of p = 0.05, none presented a pre‐specified alternative hypothesis, pre‐study power calculation and the probability of ‘false negatives’ (beta error rate). The results sections of the papers presented 49 significance tests on average (median 23, range 0–390). Of 41 studies that contained verbal descriptions of a ‘statistically non‐significant’ result, 26 (63%) falsely claimed the absence of an effect. We conclude that studies in ecology and evolutionary biology are mostly exploratory and descriptive. We should thus shift from claiming to ‘test’ specific hypotheses statistically to describing and discussing many hypotheses (possible true effect sizes) that are most compatible with our data, given our statistical model. We already have the means for doing so, because we routinely present compatibility (‘confidence’) intervals covering these hypotheses.

show abstract

Section: How Widely Are the Above Miscon Cep Ti On S Recog Nized In O...mentioning

confidence: 99%

Why and how we should join the shift from significance testing to estimation

Berner

Amrhein

2022

J of Evolutionary Biology

View full text Add to dashboard Cite

show abstract

Section: How Widely Are the Above Misconceptions Recognized In Our Li...mentioning

confidence: 99%

Why and How We Should Join the Shift From Significance Testing to Estimation

Berner¹,

Amrhein²

2022

Preprint

View full text Add to dashboard Cite

A paradigm shift away from null hypothesis significance testing seems in progress. Based on simulations, we illustrate some of the underlying motivations. First, P-values vary strongly from study to study, hence dichotomous inference using significance thresholds is usually unjustified. Second, statistically significant results have overestimated effect sizes, a bias declining with increasing statistical power. Third, statistically non-significant results have underestimated effect sizes, and this bias gets stronger with higher statistical power. Fourth, the tested statistical hypotheses usually lack biological justification and are often uninformative. Despite these problems, a screen of 48 papers from the 2020 volume of the Journal of Evolutionary Biology exemplifies that significance testing is still used almost universally in evolutionary biology. All screened studies tested default null hypotheses of zero effect with the default significance threshold of p = 0.05, none presented a pre-specified alternative hypothesis, pre-study power calculation, and the probability of ‘false negatives’(beta error). The results sections of the papers presented 49 significance tests on average (median 23, range 0–390). Of 41 studies that contained verbal descriptions of a ‘statistically non-significant’ result, 26 (63%) falsely claimed the absence of an effect. We conclude that studies in ecology and evolutionary biology are mostly exploratory and descriptive. We should thus shift from claiming to ‘test’ specific hypotheses statistically to describing and discussing many hypotheses (possible true effect sizes) that are most compatible with our data, given our statistical model. We already have the means for doing so, because we routinely present compatibility (‘confidence’) intervals covering these hypotheses.

show abstract

“…Isolated statisticians in the academic health sciences occupy a non‐traditional role as they are non‐clinicians in a medical environment, focused on team‐science research instead of independent methodological research, with less influence and generally in positions perceived as less prestigious (e.g., staff or non‐tenure track versus tenure‐track faculty). Among other contributions, statisticians also bring a commitment to rigor and reproducibility that benefits all research (Satagopan & Mazumdar, 2021; Seibold et al, 2021). The statisticians' efforts to improve the rigor and reproducibility of science more broadly with stronger study designs and enhanced data collection and analyses can potentially cause friction within a team due to a perception of divided loyalties.…”

Section: Specific Challengesmentioning

confidence: 99%

Advice for isolated statisticians collaborating in academic healthcare centre settings

Mehta

Stedman

Rao

et al. 2022

Stat

View full text Add to dashboard Cite

A substantial number of statisticians work in isolated domain science departments without access to support networks and resources typical of larger statistical units. These isolated statisticians face many challenges including limited professional networks, non‐traditional roles with idiosyncratic expectations and unique career paths. Furthermore, “the curse of success” lies ahead, as success of the department may only lead to greater demand and lagging support for the services needed. We provide suggestions for career development in an isolated environment by urging statisticians to build their network and become engaged with other statisticians to reduce isolation and identify mentors. We discuss the importance of regular communication with their supervisors to define one's role and navigate the change in expectations in the transition from a supporting role to a co‐investigator to a core leader. To help with “the curse of success,” we explain how to develop a business model with a plan for future expansion and professional development. We conclude that isolation should not be perceived as a limitation. Opportunities and resources exist within the American Statistical Association (ASA) to reduce isolation and support career development. However, there is a need for a formal career path along with evaluation criteria and example business models to support isolated statisticians striving to become core leaders.

show abstract

Statisticians, Roll Up Your Sleeves! There's A Crisis to be Solved

Abstract: Statisticians play a key role in almost all scientific research. As such, they may be key to solving the reproducibility crisis. Heidi Seibold, Alethea Charlton, Anne‐Laure Boulesteix and Sabine Hoffmann urge statisticians to take an active role in promoting more credible science

Cited by 8 publications

References 24 publications

Why and how we should join the shift from significance testing to estimation

Why and how we should join the shift from significance testing to estimation

Why and How We Should Join the Shift From Significance Testing to Estimation

Advice for isolated statisticians collaborating in academic healthcare centre settings

Contact Info

Product

Resources

About