Multisensory perception constrains the formation of object categories: a review of evidence from sensory-driven and predictive processes on categorical decisions

Abstract

Although object categorization is a fundamental cognitive ability, it is also a complex process going beyond the perception and organization of sensory stimulation. Here we review existing evidence about how the human brain acquires and organizes multisensory inputs into object representations that may lead to conceptual knowledge in memory. We first focus on evidence for two processes on object perception, multisensory integration of redundant information (e.g. seeing and feeling a shape) and crossmodal, statistical learning of complementary information (e.g. the ‘moo’ sound of a cow and its visual shape). For both processes, the importance attributed to each sensory input in constructing a multisensory representation of an object depends on the working range of the specific sensory modality, the relative reliability or distinctiveness of the encoded information and top-down predictions. Moreover, apart from sensory-driven influences on perception, the acquisition of featural information across modalities can affect semantic memory and, in turn, influence category decisions. In sum, we argue that both multisensory processes independently constrain the formation of object categories across the lifespan, possibly through early and late integration mechanisms, respectively, to allow us to efficiently achieve the everyday, but remarkable, ability of recognizing objects. This article is part of the theme issue ‘Decision and control processes in multisensory perception’.

Both stimulus‐specific and configurational features of multiple visual stimuli shape the spatial ventriloquism effect

“…However, our data show that the collective bias induced by two visual stimuli is not simply a direct sum of the biases seen when each visual stimulus is present in isolation. Rather, the collective effect is shaped by the overall configuration of the available visual information, rendering the outcome of multi-stimulus multisensory integration scene-or context-dependent, as also suggested by other work on multisensory object perception (Bertelson et al, 2000;Bizley, Maddox, & Lee, 2016;Newell et al, 2023). This result was consistently observed across three experiments that varied in the temporal onset asynchronies between the visual stimuli and the sound and the spatial discrepancies of both visual signals to the sound.…”

“…Many paradigms probe multisensory binding using paradigms involving one stimulus per modality. Whether and how results obtained using these generalize to real-life multisensory object perception remains unclear (Badde et al, 2023;Bizley, Maddox, & Lee, 2016;Duarte et al, 2023;Kirsch & Kunde, 2023;Newell et al, 2023;Tovar et al, 2020). In a step towards this direction, we asked how the presence of two visual stimuli shapes multisensory perception in the audio-visual spatial ventriloquism paradigm.…”

“…While it is known that multisensory binding is generally shaped by the spatio-temporal relation of the stimuli, their semantic attributes, their task relevance and the attentional focus (Noppeney, 2021;Shams & Beierholm, 2022;Talsma et al, 2010), still little is known about how multiple signals from one modality are combined with one signal from another. During everyday life, we usually perceive the world as consisting of various objects, and it still remains unclear how multiple signals are bound or segregated within and across modalities to facilitate a coherent interpretation of our environment (Badde et al, 2023;Duarte et al, 2023;Newell et al, 2023;Tovar et al, 2020).…”

“…modality influences multisensory perception or, more generally, how multisensory stimuli are combined during multisensory object perception (Bizley, Maddox, & Lee, 2016;Duarte et al, 2023;Newell et al, 2023;Tovar et al, 2020). While it is known that multisensory binding is generally shaped by the spatio-temporal relation of the stimuli, their semantic attributes, their task relevance and the attentional focus (Noppeney, 2021;Shams & Beierholm, 2022;Talsma et al, 2010), still little is known about how multiple signals from one modality are combined with one signal from another.…”

How the brain controls decision making in a multisensory world

“…Newell et al. [ 19 ] explore the interplay between multisensory perception and category formation across the lifespan. They argue that category formation is informed by redundant (e.g.…”