Excess significance bias in repetitive transcranial magnetic stimulation literature for neuropsychiatric disorders

Abstract

IntroductionRepetitive transcranial magnetic stimulation (rTMS) has been widely tested and promoted for use in multiple neuropsychiatric conditions, but as for many other medical devices, some gaps may exist in the literature and the evidence base for rTMS clinical efficacy remains under debate. We aimed to empirically test for an excess number of statistically significant results in the literature on rTMS therapeutic efficacy across a wide range of meta-analyses and to characterize the power of studies included in these meta-analyses. MethodsBased on power calculations, we computed the expected number of "positive" datasets for a medium effect-size (standardized mean difference, SMD=0.30) and compared it with the number of observed "positive" datasets. Sensitivity analyses considered small (SMD=0.20), modest (SMD=0.50), and large (SMD=0.80) effect sizes. Results14 meta-analyses with 228 datasets (110 for neurological disorders and 118 for psychiatric disorders) were assessed. For SMD=0.3, the number of observed "positive" studies (n=94) was larger than expected (n=35). We found evidence for an excess of significant findings overall (p<0.0001) and in 8/14 meta-analyses. Evidence for an excess of significant findings was also observed for SMD=0.5 for neurological disorders. 0 (0 %), 0 (0 %), 3 (1 %), and 53 (23 %) of the 228 datasets had power >0.80, respectively for SMDs of 0.30, 0.20, 0.50, and 0.80. ConclusionMost studies in the rTMS literature are underpowered. This results in fragmentation and waste of research efforts. The somewhat high frequency of "positive" results seems spurious and may reflect bias.

Inter‐hemispheric inhibition sculpts the output of neural circuits by co‐opting the two cerebral hemispheres

“…Indeed, even when compared only to sham stimulation, there is no demonstrable benefit of cathodal tDCS in terms of clinical measures of upper (Elsner et al 2016(Elsner et al , 2017 or lower extremity function (Elsner et al 2016). It is also apparent that with respect to rTMS in particular, and non-invasive brain stimulation in general, the results included in most meta-analyses exhibit a prior 'significance bias' , such that the true effect sizes are likely to be very much smaller than those suggested by this method of aggregation (Amad et al 2019). This inference is corroborated by the outcomes of an adequately powered multi-centre randomized controlled trial (ß200 participants) in which 15 min of 1 Hz rTMS (or sham rTMS) was delivered to the contralesional M1, prior to each of 18 sessions of rehabilitation therapy undertaken during an intervention period of 6 weeks.…”

Detecting publication selection bias through excess statistical significance

“…Thus, another application of excess significance tests is to use them to evaluate broader collections of research where there are many meta-analyses. [58][59][60] There are other reasons to prefer these tests of ESS over the Egger test and 3PSM. Because the Egger test looks for a correlation between reported effect size and its SE, some have argued that it may be a test for a smallstudy effect (perhaps even a benign effect) rather than publication selection bias.…”

Is rTMS Ready for Primetime?

“…Thus, overall, because the expected outcome is vague, poor standards of assessing response and remission are being utilized in rTMS research and regarded as acceptable for clinical use as compared to standards for pharmacological randomized controlled trials. 9,13…”

“…Due to a lower standard than that applied to novel pharmacotherapies, the evidence for the acute efficacy of rTMS as first line treatment for patients who have failed at least 1 antidepressant, is somewhat skewed and of limited quality. 9,13,14 Furthermore, this problem cannot be overcome by meta-analyses. 13…”