Where are multisensory signals combined for perceptual decision-making?

Abstract

Multisensory integration is observed in many subcortical and cortical locations including primary and non-primary sensory cortex, and higher cortical areas including frontal and parietal cortex. During unisensory perceptual tasks many of these same brain areas show neural signatures associated with decision-making. It is unclear whether multisensory representations in sensory cortex directly inform decision-making in a multisensory task, or if cross-modal signals are only combined after the accumulation of unisensory evidence at a final decision-making stage in higher cortical areas. Manipulations of neuronal activity are required to establish causal roles for given brain regions in multisensory perceptual decision-making, and so far indicate that distributed networks underlie multisensory decision-making. Understanding multisensory integration requires synthesis of small-scale pathway specific and large-scale network level manipulations.

Sounds facilitate visual motion discrimination via the enhancement of late occipital visual representations

“…In particular, many studies have argued that behaviourally relevant multisensory interactions can occur around primary-like sensory cortices and at very early latencies relative to stimulus onset (Ibrahim et al, 2016, Murray et al, 2016, Schroeder and Foxe, 2005, van Atteveldt et al, 2014). However, recent studies suggest that there may be no generic answer to this question, as multisensory processing likely involves a distributed set of task- and function-specific regions (Bizley et al, 2016, Werner and Noppeney, 2010). In line with this hypothesis, two recent fMRI studies have illustrated how the computational nature of Audio-visual interactions changes from low-level sensory to high-level parietal cortices (Rohe and Noppeney, 2014, Rohe and Noppeney, 2016).…”

“…While there is an emerging consensus that the underlying neural correlates likely involve multiple stages of the sensory decision making pathways, it remains a challenge to uncover the dynamic processes that implement the multisensory benefit for an upcoming decision in the human brain (Bizley et al, 2016, Kayser and Shams, 2015, Rohe and Noppeney, 2014, Rohe and Noppeney, 2016). For example, many studies have shown that judgements about visual motion can be influenced by simultaneous sounds (Alais and Burr, 2004, Beer and Roder, 2004, Lewis and Noppeney, 2010, Schmiedchen et al, 2012) or vestibular information (Fetsch et al, 2010, Gu et al, 2008), even so when the multisensory stimulus is not directly task relevant (Gleiss and Kayser, 2014b, Kim et al, 2012, Sekuler et al, 1997).…”

“…In particular, congruent multisensory evidence enhances visual motion discrimination performance over incongruent multisensory information (Meyer and Wuerger, 2001, Meyer et al, 2005, Soto-Faraco et al, 2003, Soto-Faraco et al, 2002). Yet, it remains difficult to attribute these perceptual benefits to specific neural processes, such as the encoding of visual motion in occipital cortices, the transformation of sensory representations into a motor response in parieto-frontal regions, or to more unspecific changes in sensory-response gain such as attentional effects (Beer and Roder, 2004, Bizley et al, 2016, Lewis and Noppeney, 2010, Talsma et al, 2010). …”

“…For example, while studies using planar motion have implied the hMT complex (but see (Baumann and Greenlee, 2007)), a study on motion in depth has pointed to a role of area V3A (Ogawa and Macaluso, 2013) and regions within the IPS (Guipponi et al, 2013). Given the frequent focus on mapping activations rather than sensory representations (Kriegeskorte et al, 2006), and given that many prior studies have relied on the relatively slow fMRI-BOLD response, these studies do not provide a detailed understanding of where and when during a trial perceptually relevant multisensory benefits emerge and are transformed into perceptual benefits on a single trial basis (Bizley et al, 2016, Zhang et al, 2016). …”

Multisensory stimuli improve relative localisation judgments compared to unisensory auditory or visual stimuli

“…Where and how multisensory signals are integrated for decision-making is likely to be task and stimulus dependent (Bizley et al , 2016). The improvement in performance observed for multisensory stimuli could arise through multiple mechanisms.…”