Category-Selective Representation of Relationships in the Visual Cortex

Abstract

Understanding social interaction requires processing social agents and their relationship. Latest results show that much of this process is visually solved: visual areas can represent multiple people encoding emergent information about their interaction that is not explained by the response to the individuals alone. A neural signature of this process is an increased response in visual areas, to face-to-face (seemingly interacting) people, relative to people presented as unrelated (back-to-back). This effect highlighted a network of visual areas for representing relational information.How is this network organized?Using functional MRI, we measured brain activity of healthy female and male humans (N=42), in response to images of two faces or two (head-blurred) bodies, facing toward or away from each other. Taking thefacing>non-facingeffect as signature of relation perception, we found that relations between faces and between bodies were coded in distinct areas, mirroring the categorical representation of faces and bodies in visual cortex. Additional analyses suggest the existence of a third network encoding relations between (non-social) objects. Finally, a separate occipitotemporal network showed generalization of relational information across body, face and non-social object dyads (multivariate-pattern classification analysis), revealing shared properties of relations across categories. In sum, beyond single entities, visual cortex encodes the relations that bind multiple entities into relationships; it does so in a category-selective fashion, thus respecting a general organizing principle of representation in high-level vision. Visual areas encoding visual relational information can reveal the processing of emergent properties of social (and non-social) interaction which trigger inferential processes.Significance statementUnderstanding social interaction requires representing the actors as well as the relation between them. We show that the earliest, rudimentary representation of a social interaction is formed in visual cortex. Using fMRI on healthy adults, we measured the brain responses to two faces or two (head-blurred) bodies, and found that, beyond representing faces and bodies, the visual cortex represents their relations, distinguishing between seemingly interacting (face-to-face) and non-interacting (back-to-back) faces/bodies. Moreover, we found that information about face and body relations is represented in separate networks, in line with the general organizing principle of categorical representation in visual cortex. The brain network encoding visual relational information may represent emergent properties of interacting people, which underlie the cognitive representation of social interaction.

Influence of social and semantic contexts on phonetic encoding in naturalistic conversations

“…Thus, we rather hypothesize that the presence of social cues, such as two speakers interacting with each other, would trigger a sharpening of the content of sentences by the perceptual system, that we observed for dialogues but not monologues. This hypothesis is in line with recent findings in the domain of visual perception showing that a human body is better perceived when facing another body (i.e., such as in a social interaction) as compared to facing in an opposite direction (Abassi & Papeo, 2023; Papeo et al, 2017; Papeo & Abassi, 2019; Vestner et al, 2019, 2021).…”

Visual Preference for Socially Relevant Spatial Relations in Humans and Monkeys

“…In particular, under low visibility conditions, a human body (but not an inanimate object) is more likely to be detected and recognized when it faces toward another body than when it faces away (Papeo & Abassi, 2019;Papeo et al, 2017;Vestner et al, 2019), yielding effects that suggest an impact of social interaction on the very early-preattentive or unconscious-stages of visual perception Xu et al, 2023), up to visual memory (Ding et al, 2017;Paparella & Papeo, 2022). The behavioral advantage in processing facing people has a counterpart in neuroimaging results showing that a person facing toward (vs. away from) another evokes stronger neural activation and distinctive neural activity patterns in visual cortex (Abassi & Papeo, 2020, 2022, 2024Walbrin & Koldewyn, 2019).…”