Visual Sensory Cortices Causally Contribute to Auditory Word Recognition Following Sensorimotor-Enriched Vocabulary Training

Abstract

Despite a rise in the use of “learning by doing” pedagogical methods in praxis, little is known as to how the brain benefits from these methods. Learning by doing strategies that utilize complementary information (“enrichment”) such as gestures have been shown to optimize learning outcomes in several domains including foreign language (L2) training. Here we tested the hypothesis that behavioral benefits of gesture-based enrichment are critically supported by integrity of the biological motion visual cortices (bmSTS). Prior functional neuroimaging work has implicated the visual motion cortices in L2 translation following sensorimotor-enriched training; the current study is the first to investigate the causal relevance of these structures in learning by doing contexts. Using neuronavigated transcranial magnetic stimulation and a gesture-enriched L2 vocabulary learning paradigm, we found that the bmSTS causally contributed to behavioral benefits of gesture-enriched learning. Visual motion cortex integrity benefitted both short- and long-term learning outcomes, as well as the learning of concrete and abstract words. These results adjudicate between opposing predictions of two neuroscientific learning theories: While reactivation-based theories predict no functional role of specialized sensory cortices in vocabulary learning outcomes, the current study supports the predictive coding theory view that these cortices precipitate sensorimotor-based learning benefits.

Visual recognition of words learned with gestures induces motor resonance in the forearm muscles

“…A previous study that made use of a larger number of stimuli (30 words per condition) revealed memory enhancement for enacted words compared to non-enacted words 99 . This could potentially also be due to the significantly shorter training in the current study compared to previous studies, in which training lasted up to 5 days 91 , 99 or longer 35 . Future research should test effects of longer or shorter training periods, as well as larger sets of vocabulary in both native and foreign languages, to further probe the relationship between memory performance and muscle activity.…”

“…They expected a modulation of motor cortex excitability if a word had previously been connected to a semantically-related gesture. Gestural motor representations were found to facilitate performance in a lexical decision task (please see Mathias et al 91 for similar effects of TMS on the learning of both abstract and concrete L2 words). These findings further support the view that motor features can become a critical part of abstract word representations if the words become associated with self-performed gestures during learning.…”

Non-verbal Enrichment in Vocabulary Learning With a Virtual Pedagogical Agent

“…Are gestures particularly strong when they are performed whereas pictures deploy their effectiveness when being viewed? A transcranial magnetic stimulation study by Mathias et al (2020) might provide support for this hypothesis. The results of this study showed that when repetitive transcranial magnetic stimulation (rTMS) was applied to the bilateral posterior motor cortices participants' translation slowed down for those L2 words that had been learned with gesture enrichment, but not for words learned with picture enrichment.…”

Design and Implementation of a Gesture-Aided E-Learning Platform

“…The content catalog consisted of 64 German language words which is a subset of the words used in Mathias et al [ 28 ]. In addition to the textual representation, synthesized speech by Google’s WaveNet-based text-to-speech engine [ 29 ] was added.…”

“…For each word, a representative gesture (cf. [ 28 ]) was recorded using the full-body motion-capture system XSENS MTw Awinda [ 31 ]. After recording, each gesture was exported into an FBX file to be suitable for animating the avatar.…”