Inside Speech: Multisensory and Modality-specific Processing of Tongue and Lip Speech Actions

Abstract

Abstract■ Action recognition has been found to rely not only on sensory brain areas but also partly on the observer's motor system. However, whether distinct auditory and visual experiences of an action modulate sensorimotor activity remains largely unknown. In the present sparse sampling fMRI study, we determined to which extent sensory and motor representations interact during the perception of tongue and lip speech actions. Tongue and lip speech actions were selected because tongue movements of our interlocutor are accessible via their impact on speech acoustics but not visible because of its position inside the vocal tract, whereas lip movements are both "audible" and visible. Participants were presented with auditory, visual, and audiovisual speech actions, with the visual inputs related to either a sagittal view of the tongue movements or a facial view of the lip movements of a speaker, previously recorded by an ultrasound imaging system and a video camera. Although the neural networks involved in visual visuo-lingual and visuo-facial perception largely overlapped, stronger motor and somatosensory activations were observed during visuo-lingual perception. In contrast, stronger activity was found in auditory and visual cortices during visuo-facial perception. Complementing these findings, activity in the left premotor cortex and in visual brain areas was found to correlate with visual recognition scores observed for visuo-lingual and visuo-facial speech stimuli, respectively, whereas visual activity correlated with RTs for both stimuli. These results suggest that unimodal and multimodal processing of lip and tongue speech actions rely on common sensorimotor brain areas. They also suggest that visual processing of audible but not visible movements induces motor and visual mental simulation of the perceived actions to facilitate recognition and/or to learn the association between auditory and visual signals. ■

Phonetic versus spatial processes during motor‐oriented imitations of visuo‐labial and visuo‐lingual speech: A functional near‐infrared spectroscopy study

“…Studies have shown that learners in general have more difficulty reading visuo‐lingual speech than visuo‐labial speech (Badin et al, 2010; Cleland et al, 2013; Grauwinkel et al, 2007; Kroger et al, 2008; Wik & Engwall, 2008). In addition, it was also found that passive perception of visuo‐lingual speech elicited greater activations in the SPL than visuo‐labial speech (Treille et al, 2017). Given the role of the SPL in action‐related spatial processing, this finding may have reflected a greater reliance on spatial information in trying to comprehend visuo‐lingual actions, which was at least partially attributable to the lack of visual familiarity with tongue actions.…”

“…In addition, the superior parietal lobe (SPL) is also involved in action comprehension with respect to understanding the planning of effector movements in space (Kawashima et al, 1995; Swinnen et al, 2010; Watanabe et al, 2004), especially when the actions are unfamiliar to the observer (Buccino et al, 2004). Functional magnetic resonance imaging (fMRI) studies found that the SPL was commonly activated during hand reaching actions towards visually perceived targets in space (Filimon et al, 2007, 2009; Kawashima et al, 1995, 1996; Roland et al, 1980; Vingerhoets, 2014) but was also activated during speech production (Grabski et al, 2012; Sato et al, 2015; Watanabe et al, 2004) and visual speech perception (Treille et al, 2017).…”

“…However, it has been suggested that visuo‐lingual speech may access phonetic information in an indirect way through an internal model of speech production that is associated with auditory speech information (D'Ausilio et al, 2014; Treille et al, 2017). In an functional magnetic resonance imaging study It has been found that while visuo‐lingual and visuo‐labial speech perception activated a common speech sensorimotor network, visuo‐lingual speech perception had greater activations in the premotor cortex and parietal regions such as the primary somatosensory cortex, IPL, SPL, and intraparietal sulcus (Treille et al, 2017).…”

In need of review: Developing Sensory Provision in Northern Ireland’s Mainstream Primary classrooms

“…Sensory Processing Disorder (SPD) has since received diagnostic criteria (Miller et al, 2005; PDM Task Force, 2006; Zero to Three, 2005) and inclusion in the DSM‐V as a sub‐set criterion in the diagnosis of Autistic Spectrum Disorder (ASD) (APA, 2013). This status has influenced many studies which link SPD with Special Education Needs and Disabilities (SEND) provision: ASD (Miguel et al, 2017; Tavassoli et al, 2018; Thye et al, 2018); Attention Deficit and Hyperactivity Disorder (ADHD) (Ghanizadeh, 2013; Shimizu et al, 2014; Jung et al, 2014); Motor difficulties (Allen and Casey, 2017; Gomez and Sirigu, 2015; Shafer et al 2017; Siaperas et al, 2014); Speech and Language difficulties (Treille et al, 2017); Tourette’s Syndrome ( Jewers et al, 2013 ) and Down’s Syndrome (van Jaarsveld et al, 2016). Further research has considered the effects upon intelligence and academic performance in the core curricular areas (Anguera et al, 2017; Zentall et al, 2013) and identified correlation between SPD and Self Injurious Behaviours (Gomez and Sirigu, 2015; Summer et al, 2017), anxiety (McDonnell et al, 2015), depression (Serafini et al, 2017) and attachment disorders (Meredith et al, 2016).…”

Rehabilitation of speech disorders following glossectomy, based on ultrasound visual illustration and feedback

“…The link between perception and production, that is to say the perceptuo-motor nature of speech as described by Schwartz, Basirat et al (2012), is made explicit in various studies that propose internal models of speech representation. For instance, Treille, Vilain et al (2016) suggest that the visual processing of audible but not visible movements (such as tongue movements) induces 'motor and visual mental simulation of the perceived actions to facilitate their recognition'.…”