Socially meaningful visual context either enhances or inhibits vocalisation processing in the macaque brain

Abstract

Social interactions rely on the interpretation of semantic and emotional information, often from multiple sensory modalities. Nonhuman primates send and receive auditory and visual communicative signals. However, the neural mechanisms underlying the association of visual and auditory information based on their common social meaning are unknown. Using heart rate estimates and functional neuroimaging, we show that in the lateral and superior temporal sulcus of the macaque monkey, neural responses are enhanced in response to species-specific vocalisations paired with a matching visual context, or when vocalisations follow, in time, visual information, but inhibited when vocalisation are incongruent with the visual context. For example, responses to affiliative vocalisations are enhanced when paired with affiliative contexts but inhibited when paired with aggressive or escape contexts. Overall, we propose that the identified neural network represents social meaning irrespective of sensory modality.

Macaque amygdala, claustrum and pulvinar support the cross-modal association of social audio-visual stimuli based on meaning

“…visual only and auditory only contexts, in the social context. This was similar to what we previously described in the superior temporal sulcus in Froesel et al (2022), suggesting a functional interaction between these two regions.…”

“…Activation patterns in the pulvinar were not predicted based on prior research and shed new light on the functional organization of this subcortical nucleus. The medial part of the nucleus and the amygdala demonstrate multisensory integration similar to what we report in the superior temporal sulcus in a previous study using the same data (Froesel et al, 2022). It is worth noting that the amygdala, claustrum and pulvinar correspond to the three subcortical regions activated by face patch stimulations.…”

“…In addition, they both are also activated by emotional facial expressions during a pure visual task. This pattern suggests that both structures contribute to context-based social and emotional processing in coordination with the face and voice patches described in Froesel et al (2022).…”

“…Exemplars from this stimulus set have already been used in several imaging studies (Belin et al, 2007; Cohen et al, 2007; Romanski, 2012; Romanski et al, 2005; Russ et al, 2008). As in our previous study (Froesel et al, 2022), all stimuli were normalized in luminance and colour but the frequency ranges varied between the different types of stimuli as shown in the supplementary Figure 4. For each of the three vocalisation categories, we used 10 unique exemplars coming from matched male and female individuals for each category in order to control for possible gender, social hierarchy or individual effects.…”

“…In a previous study, we demonstrate thanks to cardiac recordings (Froesel et al, 2020) and functional magnetic resonance imaging (fMRI) that macaques associate communicatively salient audio-visual information based on the social context set by visual information. This process is mediated by a network of face and voice patches in the superior temporal sulcus and lateral sulcus (Froesel et al, 2022). Here, using the same task and data, we identify the relevant subcortical structures in cross-modal association.…”