Cross-modal interactions between human faces and voices involved in person recognition

Joassin, Frédéric; Pesenti, Mauro; Maurage, Pierre; Verreckt, Emilie; Bruyer, Raymond; Campanella, Salvatore

doi:10.1016/j.cortex.2010.03.003

Cited by 103 publications

(78 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This information may either be associated directly, in an information-driven manner (e.g., through shared, redundant information) that is reinforced through brief learning (e.g., von Kriegstein et al, 2008), or indirectly through feedback via a common resource (e.g., Naci, Taylor, Cusack, & Tyler, 2012;ten Oever et al, 2016). For example, based on the results from a neuroimaging study, Joassin et al (2011) reported that interactions between voices and faces (static images) is underpinned by activation in supramodal regions of the brain, such as the angular gyrus and hippocampus, that can influence processing in unimodal regions of the brain associated with face and voice perception. It is interesting to speculate, however, at what point during information processing for person recognition these interactions between faces and voices arise.…”

Section: Discussionmentioning

confidence: 99%

“…For example, using probabilistic tractography, Blank, Anwander, and von Kriegstein (2011) reported evidence of direct structural connections between the fusiform face area and voice-sensitive regions of the STS. Furthermore, Joassin et al (2011) used fMRI to measure cortical activation to voices, faces and combinations of voices and faces. Their findings, that voice-face interactions result in greater activation in regions of the brain including the fusiform gyrus than either voice or face alone, are consistent with those of Blank et al (2011).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Crossmodal priming of unfamiliar faces supports early interactions between voices and faces in person perception

Bülthoff

Newell

2017

Visual Cognition

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

Crossmodal priming of unfamiliar faces supports early interactions between voices and faces in person perception

Bülthoff

Newell

2017

Visual Cognition

View full text Add to dashboard Cite

“…For instance, we are able to integrate the auditory information of what is said and the visual information of who is saying it, so that we can attribute a particular speech to a particular person (Kerlin et al, 2010) and thus take part in a conversation. In view of the obvious importance of these crossmodal processes, many studies have investigated their behavioural and cerebral correlates among healthy participants, notably leading to the identification of several brain areas dedicated to multisensory integration (Joassin et al, 2011a(Joassin et al, , 2011bLove et al, 2011). The exploration of crossmodal mechanisms thus constitutes an established field in the experimental psychology and neuroscience domains (Amedi et al, 2005;Calvert et al, 2001;De Gelder & Bertelson, 2003) and has now come to maturity, as illustrated by the proposal of integrative models (e.g.…”

mentioning

confidence: 99%

“…It is now well established that the auditory-visual integration of human faces and voices during the multimodal processing of identity and gender is associated with the activation of a specific network of cortical and subcortical regions. This network includes several regions devoted to the different cognitive processing implied in face and voice categorization task, notably (a) the unimodal visual and auditory regions processing the perceived faces and voices, which are inter-connected via a subcortical relay located in the striatum, (b) the left superior parietal gyrus, part of a larger parieto-motor network dispatching the attentional resources to the visual and auditory modalities, and (c) the right inferior frontal gyrus sustaining the integration of the semantically congruent information into a coherent multimodal representation (Joassin et al, 2011a;2011b).…”

mentioning

confidence: 99%

Towards a Crossmodal Exploration of Cognitive Deficits in Psychopathology

Maurage

Campanella

2014

Psychologica Belgica

View full text Add to dashboard Cite

“…Lesion and neuroimaging studies have suggested potential candidate cortical areas for the PINs, including the precuneus 8 , parietal and hippocampal regions [9][10][11] , posterior superior temporal sulcus (pSTS) 12,13 or the anterior temporal lobes 14 . However a PIN, as defined in Bruce & Young (1986), would correspond to a patient with a brain Scientific RepoRts | 6:37494 | DOI: 10.1038/srep37494 lesion preserving recognition and feeling of familiarity based on single modalities separately but who could not retrieve semantic information on the person, and not associate the face and voice of the person; such a patient has not yet been identified 1 .…”

mentioning

confidence: 99%

“Hearing faces and seeing voices”: Amodal coding of person identity in the human brain

Belin

Hasan

Groß

et al. 2016

International Journal of Psychophysiology

View full text Add to dashboard Cite

Recognizing familiar individuals is achieved by the brain by combining cues from several sensory modalities, including the face of a person and her voice. Here we used functional magnetic resonance (fMRI) and a whole-brain, searchlight multi-voxel pattern analysis (MVPA) to search for areas in which local fMRI patterns could result in identity classification as a function of sensory modality. We found several areas supporting face or voice stimulus classification based on fMRI responses, consistent with previous reports; the classification maps overlapped across modalities in a single area of right posterior superior temporal sulcus (pSTS). Remarkably, we also found several cortical areas, mostly located along the middle temporal gyrus, in which local fMRI patterns resulted in identity "cross-classification": vocal identity could be classified based on fMRI responses to the faces, or the reverse, or both. These findings are suggestive of a series of cortical identity representations increasingly abstracted from the input modality.• Local patterns of cerebral activity measured with fMRI can classify familiar faces or voices.• Overlap of face-and voice-classifying areas in right posterior STS.• Cross-classification of facial and vocal identity in several temporal lobe areas.The ability to recognize familiar individuals is of high importance in our social interactions. The human brain achieves this by making use of cues from several sensory modalities, including visual signals from the face of a person and auditory signals from her voice 1,2 . There is evidence that these cues are combined across senses to yield more accurate, more robust representations of person identity-a clear case of multisensory integration 3-5 . For instance, familiar speaker recognition is faster and more accurate when the voice is paired with a time-synchronized face from the same individual than when presented alone, and slower and less accurate when paired with the face of a different individual 3 . The contribution of different sensory modalities to person perception is acknowledged in particular by cognitive models such as Bruce and Young (1986)'s model of face perception. Specifically they proposed the notion of "person identity nodes" (PINs): a portion of associative memory holding identity-specific semantic codes that can be accessed via the face, the voice or other modalities: it is the point at which person recognition, as opposed to face recognition, is achieved 6,7 . Whether the PINs have a neuronal counterpart in the human brain remains unclear, in part owing to the fact that most studies of person recognition-either using neuropsychological assessment of patients with brain lesions, or neuroimaging techniques such as functional magnetic resonance imaging (fMRI) in healthy volunteers-have focused on single modality, mostly face, then, far second, voice; only few studies have investigated the cerebral bases of person recognition based on more than one sensory modality 1,4

show abstract

Cross-modal interactions between human faces and voices involved in person recognition

Cited by 103 publications

References 67 publications

Crossmodal priming of unfamiliar faces supports early interactions between voices and faces in person perception

Crossmodal priming of unfamiliar faces supports early interactions between voices and faces in person perception

Towards a Crossmodal Exploration of Cognitive Deficits in Psychopathology

“Hearing faces and seeing voices”: Amodal coding of person identity in the human brain

Contact Info

Product

Resources

About