Worse than expected: A z-curve reanalysis of motor cortex stimulation studies of embodied language comprehension

Abstract

Dozens of TMS and tDCS studies suggest a functional involvement of motor cortex in action language comprehension, supporting the embodied cognition view. In a recent study (Solana & Santiago, 2022, Neurosci. Biobehav. Rev., 141, 104834), we evaluated the soundness of this literature by means of p-curve analyses and tests for excess significance. The analysis estimated a low average power (≈ 30%) and showed signs of publication bias, which led us to conclude that this body of findings does not stand on solid ground. Yet, the employed techniques seem to perform poorly when high heterogeneity is present (as it is the case) and cannot quantify the amount of publication bias. For these reasons, in the present commentary, we reanalyzed the same set of studies (N = 43) with a method that does not have these limitations: z-curve analysis. Z-curve not only replicated our prior conclusions but showed an even more heartbreaking situation, with a lower power estimation (≈ 20%) and clear signs of a strong publication bias (around 7-10 unpublished contrasts expected to exist for each published one). Researchers on this topic should consider these shortcomings before drawing conclusions from this body of findings, as well as start implementing robust and transparent research practices to improve its reliability.

No support for a causal role of primary motor cortex in construing meaning from language: An rTMS study

“…As a final example, we recently meta-analyzed 43 TMS and tDCS studies of motor cortex 6 stimulation during action language comprehension by means of p-curve analyses and tests for excess significance (Solana and Santiago, 2022), as well as z-curve analyses (Solana and Santiago, 2023). Our results suggested that (1) we cannot conclude that these studies explore real effects, (2) their estimated underlying power is quite low (i.e., the majority of them may not replicate if repeated identically), and (3) they contain clear signs of a large publication bias.…”