Justify Your Alpha

Lakens, Daniël; Adolfi, Federico; Albers, Casper J.; Anvari, Farid; Apps, Matthew A J; Argamon, Shlomo; Assen, Marcel A. L. M. van; Baguley, Thom; Becker, Raymond; Benning, Stephen D.; Bradford, Daniel E.; Buchanan, Erin Michelle; Caldwell, Aaron R.; Calster, Ben Van; Carlsson, Rickard; Chen, Sau-Chin; Chung, Bryan; Colling, Lincoln; Collins, Gary S.; Crook, Zander; Cross, Emily S.; Daniels, Sameera; Danielsson, Henrik; DeBruine, Lisa M.; Dunleavy, Daniel J.; Earp, Brian D.; feist, michele; Ferrell, Jason D.; Field, James G.; Fox, Nicholas William; Friesen, Amanda; Gomes, Caio; Grange, James A.; Grieve, Andrew P.; Guggenberger, Robert; Harmelen, Anne‐Laura van; Hasselman, Fred; Hochard, Kevin D.; Hoffarth, Mark R.; Holmes, Nicholas P.; Ingre, Michael; Isager, Peder Mortvedt; Isotalus, Hanna K.; Johansson, Christer; Juszczyk, Konrad; Kenny, David A.; Khalil, Ahmed A.; Konat, Barbara; Lao, Junpeng; Larsen, Erik Gahner; Lodder, Gerine M. A.; Lukavský, Jiřı́; Madan, Christopher R.; Manheim, David; González-Márquez, Mónica; Martin, Stephen R.; Martin, Andrea E.; Mayo, Deborah G.; McCarthy, Randy J.; McConway, Kevin James; McFarland, Colin; Nilsonne, Gustav; Nio, Amanda Q. X.; Oliveira, Cilene Lino de; Parsons, Sam; Pfuhl, Gerit; Quinn, Kimberly A.; Sakon, John J.; Sarıbay, S. Adil; Schneider, Iris K.; Selvaraju, Manojkumar; Sjoerds, Zsuzsika; Smith, S. Ray; Smits, Tim; Spies, Jeffrey R.; Sreekumar, Vishnu; Steltenpohl, Crystal N.; Stenhouse, Neil; Świątkowski, Wojciech; Vadillo, Miguel A.; Williams, Matt; Williams, Samantha E.; Williams, Donald R.; Xivry, Jean-Jacques Orban de; Yarkoni, Tal; Ziano, Ignazio; Zwaan, Rolf Antonius

doi:10.31234/osf.io/9s3y6

Cited by 163 publications

(200 citation statements)

References 6 publications

Supporting

Mentioning

193

Contrasting

Unclassified

Order By: Relevance

“…On top of this, studies that examine effects of training or other types of interventions that take place across time are often searching for subtle effects that are even more elusive than what is being examined during single-session fMRI studies [69]. While we absolutely acknowledge that a larger sample size would have been ideal to test the current hypotheses, we also argue that timeintensive training invention studies, such as the one described here, are valuable for understanding how cognitive and perceptual processes change across time, and are worth considering even if there are power considerations [70]. Future studies seeking to examine how socialising with artificial agents shapes social perception and cognitive processes may wish to deploy longer or more intensive socialising or training procedures, as well as experiment with alternative methods for examining the neurocognitive mechanisms supporting social engagement with human and robotic agents.…”

Section: General Limitations and Ideas For Future Researchmentioning

confidence: 82%

A neurocognitive investigation of the impact of socialising with a robot on empathy for pain

Cross

Riddoch

Pratts

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

To what extent can humans form social relationships with robots? In the present study, we combined functional neuroimaging with a robot socialising intervention to probe the flexibility of empathy, a core component of social relationships, toward robots. Twenty-six individuals underwent identical fMRI sessions before and after being issued a social robot to take home and interact with over the course of a week. While undergoing fMRI, participants observed videos of a human actor or a robot experiencing pain or pleasure in response to electrical stimulation. Repetition suppression of activity in the pain network, a collection of brain regions associated with empathy and emotional responding, was measured to test whether socialising with a social robot leads to greater overlap in neural mechanisms when observing human and robotic agents experiencing pain or pleasure. In contrast to our hypothesis, functional region-of-interest analyses revealed no change in neural overlap for agents after the socialising intervention. Similarly, no increase in activation when observing a robot experiencing pain emerged post-socialising. Whole-brain analysis showed that, before the socialising intervention, superior parietal and early visual regions are sensitive to novel agents, while after socialising, medial temporal regions show agent sensitivity. A region of the inferior parietal lobule was sensitive to novel emotions, but only during the presocialising scan session. Together, these findings suggest that a short socialisation intervention with a social robot does not lead to discernible differences in empathy toward the robot, as measured by behavioural or brain responses. We discuss the extent to which longer term socialisation with robots might shape social cognitive processes and ultimately our relationships with these machines.

show abstract

Section: General Limitations and Ideas For Future Researchmentioning

confidence: 82%

A neurocognitive investigation of the impact of socialising with a robot on empathy for pain

Cross

Riddoch

Pratts

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recently, some have suggested that researchers choose (and justify) an "optimal" value for α, for each unique study; see Mudge et al (2012), Ioannidis et al (2013) and Lakens et al (2018). Each study within a journal would thereby have a different set of criteria.…”

Section: Resultsmentioning

confidence: 99%

The World of Research Has Gone Berserk: Modeling the Consequences of Requiring “Greater Statistical Stringency” for Scientific Publication

Campbell

Gustafson

2019

The American Statistician

View full text Add to dashboard Cite

In response to growing concern about the reliability and reproducibility of published science, researchers have proposed adopting measures of 'greater statistical stringency', including suggestions to require larger sample sizes and to lower the highly criticized 'p < 0.05' significance threshold. While pros and cons are vigorously debated, there has been little to no modeling of how adopting these measures might affect what type of science is published. In this paper, we develop a novel optimality model that, given current incentives to publish, predicts a researcher's most rational use of resources in terms of the number of studies to undertake, the statistical power to devote to each study, and the desirable pre-study odds to pursue. We then develop a methodology that allows one to estimate the reliability of published research by considering a distribution of preferred research strategies. Using this approach, we investigate the merits of adopting measures of 'greater statistical stringency' with the goal of informing the ongoing debate.

show abstract

“…These authors argue that a change in alpha would, among other things, reduce the number of "false positives" that can contribute to irreproducible results with a significance level of p ≤ 0.05. However, another equally large and eminent group of statisticians (9) disagrees with this recommendation, enumerating drawbacks to a much tighter significance level, including an increase in the "false negative rate," i.e. missing out on a genuine discovery.…”

Section: Discussionmentioning

confidence: 99%

Hypothesis-Testing Improves the Predicted Reliability of Neuroscience Research

2019

Preprint

View full text Add to dashboard Cite

Critics often cite statistical problems as prime contributors to the "reproducibility crisis" of science, expressing great concern about research that bases major conclusions on single pvalued statistical tests. The critics also argue that the predicted reliability of neuroscience research in particular is low because much of the work depends heavily on small experimental sample sizes and, hence, its statistical tests lack adequate "power." It isn't known how common the practice of basing major conclusions on single tests is in neuroscience or how the statistical criticisms affect the validity of conclusions drawn by laboratory research that evaluates hypotheses via multiple tests. I review a sample of neuroscience publications to estimate the prevalence and extensiveness of hypothesis-testing research. I then apply R.A. Fisher's method for combining test results to show that the practice of testing multiple predictions of hypotheses increases the predicted reliability of neuroscience research.

show abstract

Justify Your Alpha

Abstract: In response to recommendations to redefine statistical significance to p ≤ .005, we propose that researchers should transparently report and justify all choices they make when designing a study, including the alpha level.

Cited by 163 publications

References 6 publications

A neurocognitive investigation of the impact of socialising with a robot on empathy for pain

A neurocognitive investigation of the impact of socialising with a robot on empathy for pain

The World of Research Has Gone Berserk: Modeling the Consequences of Requiring “Greater Statistical Stringency” for Scientific Publication

Hypothesis-Testing Improves the Predicted Reliability of Neuroscience Research

Contact Info

Product

Resources

About