Selective enhancement of object representations through multisensory integration

Abstract

1Objects are the fundamental building blocks of how we create a representation of the external 2 world. One major distinction amongst objects is between those that are animate versus 3 inanimate. Many objects are specified by more than a single sense, yet the nature by which 4 multisensory objects are represented by the brain remains poorly understood. Using 5 representational similarity analysis of human EEG signals, we show enhanced encoding of 6 audiovisual objects when compared to their corresponding visual and auditory objects. 7Surprisingly, we discovered the often-found processing advantages for animate objects was not 8 evident in a multisensory context due to greater neural enhancement of inanimate objects-the 9 more weakly encoded objects under unisensory conditions. Further analysis showed that the 10 selective enhancement of inanimate audiovisual objects corresponded with an increase in shared 11representations across brain areas, suggesting that neural enhancement was mediated by 12 multisensory integration. Moreover, a distance-to-bound analysis provided critical links between 13 neural findings and behavior. Improvements in neural decoding at the individual exemplar level 14 for audiovisual inanimate objects predicted reaction time differences between multisensory and 15 unisensory presentations during a go/no-go animate categorization task. Interestingly, links 16 between neural activity and behavioral measures were most prominent 100 to 200ms and 350 to 17 500ms after stimulus presentation, corresponding to time periods associated with sensory 18 evidence accumulation and decision-making, respectively. Collectively, these findings provide 19 key insights into a fundamental process the brain uses to maximize information it captures across 20 sensory systems to perform object recognition. Significance Statement 1Our world is filled with an ever-changing milieu of sensory information that we are able to 2 seamlessly transform into meaningful perceptual experience. We accomplish this feat by 3 combining different features from our senses to construct objects. However, despite the fact that 4 our senses do not work in isolation but rather in concert with each other, little is known about 5 how the brain combines the senses together to form object representations. Here, we used EEG 6 and machine learning to study how the brain processes auditory, visual, and audiovisual objects. 7Surprisingly, we found that non-living objects, the objects which were more difficult to process 8 with one sense alone, benefited the most from engaging multiple senses.

Visual Influences on Auditory Behavioral, Neural, and Perceptual Processes: A Review

“…There are a number of ways to address these issues. One is to use decoding methods to abstract the amount of information present in a multisensory signal when compared with unisensory signals (Jung et al 2018;Tovar et al 2020). Here, the added activation will only be beneficial if it carries added unique information from each sensory modality.…”

Towards understanding how we pay attention in naturalistic visual search settings

“…Compared to semantically incongruent or meaningless (arbitrary) multisensory stimuli, semantically congruent stimuli are more easily identified and remembered (e.g. Laurienti et al 2004; Murray et al, 2004; Doehrmann & Naumer 2008; Chen & Spence, 2010; Matusz et al, 2015a; Tovar et al 2020; reviewed in ten Oever et al 2016; Murray et al, 2016b; Matusz et al 2020) and also, more strongly attended (Matusz et al 2015b, 2019a, 2019b; reviewed in Soto-Faraco et al, 2019; Matusz et al 2019c). For example, Iordanenscu et al (2009) demonstrated that search for naturalistic objects is faster when accompanied by irrelevant albeit congruent sounds.…”