Principles of Multisensory Behavior

Abstract

The combined use of multisensory signals is often beneficial. Based on neuronal recordings in the superior colliculus of cats, three basic rules were formulated to describe the effectiveness of multisensory signals: the enhancement of neuronal responses to multisensory compared with unisensory signals is largest when signals occur at the same location ("spatial rule"), when signals are presented at the same time ("temporal rule"), and when signals are rather weak ("principle of inverse effectiveness"). These rules are also considered with respect to multisensory benefits as observed with behavioral measures, but do they capture these benefits best? To uncover the principles that rule benefits in multisensory behavior, we here investigated the classical redundant signal effect (RSE; i.e., the speedup of response times in multisensory compared with unisensory conditions) in humans. Based on theoretical considerations using probability summation, we derived two alternative principles to explain the effect. First, the "principle of congruent effectiveness" states that the benefit in multisensory behavior (here the speedup of response times) is largest when behavioral performance in corresponding unisensory conditions is similar. Second, the "variability rule" states that the benefit is largest when performance in corresponding unisensory conditions is unreliable. We then tested these predictions in two experiments, in which we manipulated the relative onset and the physical strength of distinct audiovisual signals. Our results, which are based on a systematic analysis of response time distributions, show that the RSE follows these principles very well, thereby providing compelling evidence in favor of probability summation as the underlying combination rule.

Visual-somatosensory integration in aging: Does stimulus location really matter?

“…Nonetheless, tests developed to assess race model violations are inherently controversial (Eriksen et al, 1989; Mordkoff & Yantis, 1991) since these models are conservative (Miller, 1986; Gondan et al, 2004) and have inherent limitations like assumptions about independence between unisensory processes (Colonius & Diederich, 2006; see also Mahoney et al, 2011). Similarly, work from Otto and Mamassian (2012) and Otto et al (2013) also reveals that RT facilitation can be explained by probability summation, providing evidence for multisensory integrative effects even when distinct parallel processes are not integrated.…”

“…Similarly and as part of a larger study, Teder-Sälejärvi et al (2005) reported statistically significant RT facilitation to multisensory AV conditions regardless of spatial alignment. Collectively, these results revealed that AV and AS interactions in humans are not constrained by space using a simple RT task; however, other researchers argue that the spatial rule fails in some cases because behavioral analyses assessing redundant signals are not sensitive to spatial alignment (see Otto et al, 2013). …”

On the relative contributions of multisensory integration and crossmodal exogenous spatial attention to multisensory response enhancement

“…One must carefully consider the paradigm that was used when interpreting race model violations, given that the amount of race model violation can be affected by various factors that are perhaps not directly linked to multisensory integration. For example, response preparation and modality switch effects can also contribute to a (relative) speed up of responses to multisensory stimuli (e.g., Gondan, Lange, Rösler, & Röder, 2004;Los & Van der Burg, 2013;Otto & Mamassian, 2012;Otto, Dassy, & Mamassian, 2013;. That being said, in the redundant target effect paradigm (i.e., the integration block) race model violations have been shown to relate to neural measures of multisensory integration (e.g., Gondan, Niederhaus, Rösler, & Röder, 2005;Molholm et al, 2002Molholm et al, , 2006; also see Mercier et al, 2015).…”

A matter of attention: Crossmodal congruence enhances and impairs performance in a novel trimodal matching paradigm

“…This pattern would suggest that best performance in crossmodal matching is achieved if the modalities to be integrated have comparable levels of perceptual gain. A recent modelling approach of audio-visual redundant signal detection led to a similar conclusion and postulated that crossmodal integration can be expected to be most beneficial for signals that would individually lead to comparable performance levels(Otto et al, 2013).4.2 Crossmodal congruence and perceptual competitionCrossmodal congruence between the task-irrelevant modality and attended congruent modalities improved performance in all attentional foci with comparable magnitude. This boost in performance by irrelevant but congruent sensory information suggests that crossmodal congruence enhances perceptual processing irrespective of attention.Alternatively, this performance difference might merely be related to the absence of conflicting information in the fully congruent trials.…”

“…Although supporting evidence for the validity of these principles in human behaviour exists (e.g., Bolognini, Frassinetti, Serino, & Làdavas, 2004;Senkowski, Saint-Amour, Höfle, & Foxe, 2011), an increasing number of empirical null results and methodological issues question the general applicability of these principles (Holmes, 2007;Otto, Dassy, & Mamassian, 2013;Pannunzi et al, 2015;Spence, 2013;Sperdin, Cappe, & Murray, 2010).…”