Erroneous analyses of interactions in neuroscience: a problem of significance

Abstract

In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling."The percentage of neurons showing cue-related activity increased with training in the mutant mice (P < 0.05), but not in the control mice (P > 0.05)." "Animals receiving vehicle (control) infusions into the amygdala showed increased freezing to the conditioned stimulus compared with a control stimulus (P < 0.01); in animals receiving muscimol infusions into the amygdala, this difference was abolished (F < 1)."These two fictive, but representative, statements illustrate a statistical error that is common in the neuroscience literature. The researchers who made these statements wanted to claim that one effect (for example, the training effect on neuronal activity in mutant mice) was larger or smaller than the other effect (the training effect in control mice). To support this claim, they needed to report a statistically significant interaction (between amount of training and type of mice), but instead they reported that one effect was statistically significant, whereas the other effect was not. Although superficially compelling, the latter type of statistical reasoning is erroneous because the difference between significant and not significant need not itself be statistically significant 1 . Consider an extreme scenario in which traininginduced activity barely reaches significance in mutant mice (for example, P = 0.049) and barely fails to reach significance for control mice (for example, P = 0.051). Despite the fact that these two P values lie on opposite sides of 0.05, one cannot conclude that the training effect for mutant mice differs statistically from that for control mice.That is, as famously noted by Rosnow and Rosenthal 2 , "surely, God loves the 0.06 nearly as much as the 0.05". Thus, when making a comparison between two effects, researchers should report the statistical significance of their difference rather than the difference between their significance levels.Our impression was that this error of comparing significance levels is widespread in the neuroscience literature, but until now there were no aggregate data to support this impression. We therefore examined all of the behavioral, systems and cognitive neuroscience studies published in four prestigious journals (Nature, Science, Nature Neuroscience and Neuron) ...

A solution to dependency: using multilevel analysis to accommodate nested data

“…Serious concerns have, however, been raised about an excess of false positive results contami nating the neuroscience literature [1][2][3][4] . Controlling the false positive rate is critical, since theoretical progress in the neuroscience field relies fundamentally on drawing correct conclusions from experi mental research.…”

“…Various recent publications force neuroscientists to acknowledge the possibility that the harvest of their hard labor is contaminated by an abundance of false positive effects [1][2][3][4] . Nested designs are ubiq uitous in neuroscience, and an increased awareness of the problem of nesting in both researchers and reviewers will prevent costly and time consuming quixotic pursuits of spurious effects, thereby assist ing progress in the understanding of the nervous system.…”

Receiver operating characteristic analysis of age-related changes in lineup performance

“…Separate analyses of hit and false alarm rates do not allow for drawing conclusions about how performance differs in when the target is present versus absent. As pointed out by Gronlund and colleagues (2012), performance in target present and target absent lineups must be analysed simultaneously in order to draw inferences about memory processes (also see Gelman & Stern, 2006;Nieuwenhuis, Forstmann, & Wagenmakers, 2011). It is not correct, analytically speaking, to conclude that there are age-related performance differences when the target is absent, and not when the target is present, unless performance in target present and target absent lineups are examined in a factorial analysis.…”

Uso da estatística na Educação Física: análise das publicações nacionais entre os anos de 2009 e 2011

“…Refl exo disso pode ser observado no fato de que mesmo entre artigos publicados em periódicos internacionais de alto impacto, existe signifi cativa ocorrência de erro durante a aplicação de procedimentos estatísticos 8,10 . Estudos demonstram que a checagem de pressupostos para utilização de testes estatísticos é reportada em apenas 8-25% dos estudos 8,[11][12] .…”

“…Além disso, tal fato não ocorre devido ao conhecimento da robustez dos testes utilizados, e sim por desconhecimento dos pressupostos e de como verifi ca-los 8 . Tal panorama parece indicar que tanto os pesquisadores quanto os revisores das revistas científi cas possuem dúvidas referentes ao procedimento correto a ser adotado de acordo com os tipos de dados em questão 10,13 . Diante da importância do uso adequado da estatística em trabalhos científi cos, o objetivo do presente estudo foi verifi car o percentual de artigos científi cos publicados em periódicos nacionais da área da Educação Física que utilizam estatística, bem como quais são os principais testes utilizados e se respeitam os pressupostos básicos para utilização dos mesmos.…”