Multi-sensory integration of spatio-temporal segmentation cues: one plus one does not always equal two

Abstract

How are multiple, multi-sensory stimuli combined for use in segmenting spatio-temporal events? For an answer, we measured the effect of various auditory or visual stimuli, in isolation or in combination, on a bistable percept of visual motion ("bouncing" vs. "streaming"). To minimize individual differences, the physical properties of stimuli were adjusted to reflect individual subjects' sensitivity to each cue in isolation. When put into combination, perceptual influences that had been equipotent in isolation were substantially altered. Specifically, auditory cues that had been strong when presented alone, were greatly reduced in combination. Evaluation of alternative models of sensory integration showed that the state of the visual bistable percept could not be accounted for by probability summation among cues, as might occur at the level of decision processes. Instead, the state of the bistable percept was well predicted from a weighted sum of cues, with visual cues strongly dominating auditory cues. Finally, when cue weights were compared for individual subjects, it was found that subjects differ somewhat in the strategy they use for integrating multi-sensory information.

Auditory transients do not affect visual sensitivity in discriminating between objective streaming and bouncing events

“…The persistence of reported bouncing on sound trials in spite of progressively greater texture density differences observed in Experiment 1 is surprising in light of the modality appropriateness hypothesis (e.g., Welch & Warren, 1980;Soto-Faraco, Spence, Lloyd, & Kingstone, 2004;Heron, Whitaker, & McGraw, 2004;Zhou, Wong, & Sekuler, 2007), which posits that the resolution of a multisensory display should be determined by information to the sensory modality that provides the most accurate estimate of the attribute. Vision is assumed to provide more accurate information about the location and motion of objects than audition and therefore should determine the resolution of our unambiguous displays.…”

“…Another study by Zhou et al (2007) is relevant. These authors investigated the efficacy of two auditory cues (timing relative to the point of coincidence, auditory intensity) and two visual cues (luminance change at coincidence, pause in motion at coincidence) as well as their combinations in influencing the resolution of ambiguous stream bounce displays.…”

Noise, multisensory integration, and previous response in perceptual disambiguation

“…In contrast, the luminance of pixels that do not correspond to the moving target (i.e., the points off the diagonals in Fig 2B, left) should be darker than average for them to trigger more likely a bounce response. In other words, this indicates that light stimuli moving against a dark background are more likely perceived as bouncing (see [36]). Additionally, the luminance kernel has more energy prior to the intersection than after.…”

“…The result of such motion energy computation is then combined with auditory information and recent perceptual memory into a single proxy variable representing the overall evidence towards one interpretation or the other. This is done in the following way: In line with current models of sensory integration [36, 44], assuming neural noise to be independent across the channels and normally distributed, we modeled sensory integration of visual motion energy, auditory signals, and recent perceptual memory as weighed linear integration. To make the stream-bounce information provided by the different channels (motion energy, audio signal, memory) directly comparable, linear coefficients ω i consisted of a combination of both weights and scaling factors [31, 45].…”

“…The present results demonstrate which cues are relevant to resolve perceptual ambiguity in the stream-bounce display, and highlight the mechanisms underlying both the computation and the combination of such multisensory cues for the perception of dynamic ambiguous displays. Previous research has already investigated the stimulus properties biasing the interpretation of the stream-bounce display (e.g., [36, 38, 46]). Such studies relied on a parametric manipulation of one or more stimulus features.…”