Visual and Motor Cortices Differentially Support the Translation of Foreign Language Words

Abstract

At present, it is largely unclear how the human brain optimally learns foreign languages. We investigated teaching strategies that utilize complementary information ("enrichment"), such as pictures or gestures, to optimize vocabulary learning outcome. We found that learning while performing gestures was more efficient than the common practice of learning with pictures and that both enrichment strategies were better than learning without enrichment ("verbal learning"). We tested the prediction of an influential cognitive neuroscience theory that provides explanations for the beneficial behavioral effects of enrichment: the "multisensory learning theory" attributes the benefits of enrichment to recruitment of brain areas specialized in processing the enrichment. To test this prediction, we asked participants to translate auditorily presented foreign words during fMRI. Multivariate pattern classification allowed us to decode from the brain activity under which enrichment condition the vocabulary had been learned. The visual-object-sensitive lateral occipital complex (LOC) represented auditory words that had been learned with pictures. The biological motion superior temporal sulcus (bmSTS) and motor areas represented auditory words that had been learned with gestures. Importantly, brain activity in these specialized visual and motor brain areas correlated with behavioral performance. The cortical activation pattern found in the present study strongly supports the multisensory learning theory in contrast to alternative explanations. In addition, the results highlight the importance of learning foreign language vocabulary with enrichment, particularly with self-performed gestures.

L2 Vocabulary Teaching with Student- and Teacher-Generated Gestures: A Classroom Perspective

“…In the discussion, words learned with gestures were described, for example, as being more "sticky" and that thinking or seeing the gesture made the word "pop into your head." Although these student perceptions cannot be definitively supported within the current action research design, they lend support to the experimental research cited previously, which showed that words learned with gestures are recalled better, especially in the long term (Macedonia & Klimesch, 2014;Mayer et al, 2015;Tellier, 2008), and are in line with theoretical frameworks that posit learning benefits when perceptual and motor cues are used, such as Dual Coding Theory (Paivio, 1986) and Cognitive Load Theory (Sweller et al, 2011).…”

“…Although there are limits to when such gestures can be used-for example, gestures for similar concepts can overlap and gestures alone cannot illustrate the meanings of abstract words-studies have found that words learned in this VOLUmE 34, iSSUE 1,2016 manner are recalled better (Kelly, McDevitt, & Esch, 2009;Morett, 2014) and used more in writing (Macedonia & Knösche, 2011) than are words learned via L1 translations, potentially because adding gestures leaves deeper memory traces (Kelly et al, 2009). Learner involvement appears to be another factor: studies have shown that gestures lead to better word recall than do other visual supports, such as pictures, especially when learners actively repeat both word and gesture (Mayer, Yildiz, Macedonia, & von Kriegstein, 2015;Tellier, 2008).…”

“…However, the peer teaching task in the santé lesson did not bring about much repetition-many students repeated the gestures just once or twice, and repeated the words only two thirds of the time. Given that prior research has shown it is best for learners to repeat both words and gestures (Mayer et al, 2015) and that students themselves felt they learned best by repeating word-gesture combinations multiple times, student-generated gesture activities should be designed to ensure sufficient repetition.…”

Primary motor cortex functionally contributes to language comprehension: An online rTMS study

“…For example, handedness influences whether left or right motor cortex becomes active during action word processing (Willems et al 2010), proficiency in first or second language influences the extent of such activation (Vukovic and Shtyrov 2014), as does professional and motoric experience, such as in experts vs. novices (Del et al 2010;Behmer and Jantzen 2011). Learning context also plays a role, and activity in visual and motor cortices predicts superior performance even months after enriched (multi-sensory) word learning, but not if words are learned by passive reading or translation (Mayer et al 2015). Finally, more research is needed on how TMS affects semantically shallow tasks, in which performance does not depend on conscious retrieval of word meaning.…”

International Research on Translation Teaching: A Review (2008-2017)

“…They use Event-related Potentials (ERPs) and functional Magnetic Resonance Imaging (fMRI) techniques to explore the complex cognitive and translation process of learners. Mayer, et al (2015) investigate the teaching effects of two strategies: picture teaching and body language teaching in vocabulary translation learning. They use fMRI to decode the visual and cortical movements of the brain as students listen and translate.…”