The Time-Course of the Cross-Modal Semantic Modulation of Visual Picture Processing by Naturalistic Sounds and Spoken Words

Abstract

The time-course of cross-modal semantic interactions between pictures and either naturalistic sounds or spoken words was compared. Participants performed a speeded picture categorization task while hearing a task-irrelevant auditory stimulus presented at various stimulus onset asynchronies (SOAs) with respect to the visual picture. Both naturalistic sounds and spoken words gave rise to cross-modal semantic congruency effects (i.e., facilitation by semantically congruent sounds and inhibition by semantically incongruent sounds, as compared to a baseline noise condition) when the onset of the sound led that of the picture by 240 ms or more. Both naturalistic sounds and spoken words also gave rise to inhibition irrespective of their semantic congruency when presented within 106 ms of the onset of the picture. The peak of this cross-modal inhibitory effect occurred earlier for spoken words than for naturalistic sounds. These results therefore demonstrate that the semantic priming of visual picture categorization by auditory stimuli only occurs when the onset of the sound precedes that of the visual stimulus. The different time-courses observed for naturalistic sounds and spoken words likely reflect the different processing pathways to access the relevant semantic representations.

Multisensory enhancement elicited by unconscious visual stimuli

“…Consistent with the findings in our control group, the likelihood of auditory and visual signals being integrated into a unified percept is greatest when auditory and visual signals are spatially and temporarily aligned within 100 ms (e.g., Lewald and Guski 2003 ), nevertheless, multisensory stimuli can be integrated into unified objects or events even with temporal disparities up to 800 ms (Wallace et al 2004b ). Interestingly, similar to our case with PCA but not our control participants, using a speeded picture categorization task, Chen and Spence ( 2013 ) showed multisensory semantic congruence effects only when the auditory stimulus preceded the visual signal by 240 ms or more. In contrast, with SOAs within 100 ms an inhibitory effect was found irrespective of semantic congruence.…”

“…Differences in findings between studies could be due to the fact that we used a simple detection rather than an image categorization task. Indeed, the spatial and temporal properties of multisensory integration have a complex relationship dependent not only on the task at hand, but the type of signals being integrated (e.g., novel signals vs. naturalistic images and sounds) (e.g., Chen and Spence 2013 , 2017 ; Spence 2013 ; Stevenson et al 2012 ). In this case with PCA, given that the primary visual and posterior parietal brain regions are affected, a degradation or lack of visual information from these brain regions may have led to the broadening of the temporal integration window for multisensory signals to compensate for the loss of visual information.…”

Assessing the Role of the ‘Unity Assumption’ on Multisensory Integration: A Review

“…The Y-axis represents the spatiotemporal disparity between the stimuli. The effects of crossmodal correspondences and semantic congruency often occur between stimuli in a larger temporal disparity over hundreds of ms that are represented as two distinct events, such as the studies demonstrating crossmodal semantic priming ( Chen and Spence, 2011b , 2013 ). The unity effect attributable to crossmodal correspondences or semantic congruency has, though, only been observed when the stimuli were presented in a range within 100 ms ( Vatakis and Spence, 2007 ; Parise and Spence, 2009 ).…”

The Interplay Between Multisensory Associative Learning and IQ in Children

“…Multisensory processes underlie many stages of information processing (Calvert, ; Driver & Noesselt, ). However, the nature of the neural mechanisms involved and their developmental trajectories are largely unknown, though it is apparent that the neural mechanisms depend on the sensory signal type (learnt or novel; Chen & Spence, , ; Laine, Kwon, & Hamalainen, ; Molholm, Ritter, Javitt, & Foxe, ; Raij, Uutela, & Hari, ). Further, these mechanisms also depend on whether sensory signals are merged into a unified percept (i.e., multisensory integration) or whether specific sensory features are linked over time (e.g., transferring or matching information across the sensory systems, associative learning, etc.…”