Cross-modal integration of reward value during oculomotor planning

Abstract

Reward value guides goal-directed behavior and modulates early sensory processing. Rewarding stimuli are often multisensory but it is not known how reward value is combined across sensory modalities. Here we show that the integration of reward value critically depends on whether the distinct sensory inputs are perceived to emanate from the same multisensory object. We systematically manipulated the congruency in monetary reward values and the relative spatial positions of co-occurring auditory and visual stimuli that served as bimodal distractors during an oculomotor task. The amount of interference induced by the distractors was used as an indicator of their perceptual salience. Our results across two experiments show that when reward value is linked to each modality separately, the value congruence between vision and audition determines the combined salience of the bimodal distractors. However, reward value of vision wins over the value of audition if visual and auditory stimuli have been experienced as belonging to the same audiovisual object prior to the learning of the reward values. The perceived spatial alignment of auditory and visual stimuli is a prerequisite for the integration of their reward values, as no effect of reward value was observed when the two modalities were perceived to be misaligned. These results show that in a task that highly relies on the processing of visual spatial information, the reward values from multiple sensory modalities are integrated with each other, each with their respective weights. This weighting depends on the congruency in reward values, exposure history, and spatial co-localization.

Differential effects of intra-modal and cross-modal reward value on visual perception: ERP evidence

“…Specifically, we found a reduction in N1 negativity after learning of the reward associations, which is different from an enhanced P1 positivity and N1 negativity that has been observed in studies of cross-modal attention (Busse et al, 2005; Störmer et al, 2009; Zimmer et al, 2010). One possible mechanism is a reward-driven enhancement of audiovisual integration, which is in line with recent findings demonstrating a role of reward in multisensory integration (Bean et al, 2021; Cheng et al, 2020). This mechanism can also explain the direction of ERP modulations, as previous studies found that an audiovisual stimulus elicits response modulations of visual ERPs mainly in N185 window, with a reduction of the negativity of N185 component compared to the unimodal stimuli (Giard and Peronnet, 1999), which is similar to the pattern of modulations we observed for cross-modal rewards.…”

“…A second possibility is that cross-modal reward cues enhance visual perception by strengthening the audiovisual integration of the auditory and visual components of an audiovisual stimulus. Although audiovisual integration largely occurs automatically (for a review see Calvert and Thesen, 2004), top-down factors such as attention (Alsius et al, 2005; Navarra et al, 2010; Talsma et al, 2007) and recently reward value (Bean et al, 2021; Cheng et al, 2020) have been shown to affect its strength. Through a more efficient integration with the visual cues, auditory reward signals could hence capture attention not only to themselves but also to the whole audiovisual object including the visual target, thereby improving performance.…”

Previously reward-associated sounds interfere with goal-directed auditory processing

“…Many studies of attentional bias that utilise auditory stimuli investigate cross-modal interfacing of visual and auditory systems and focus on how the addition of sound stimuli modulates visual processing (e.g., Anderson, 2016c;McDonald et al, 2000McDonald et al, , 2005Sanz et al, 2018;Stormer et al, 2009). Cheng et al (2020) argued that the reward value of visual and audio inputs are integrated together and that the associative value of vision dominates over the associative value of audition, highlighting a need to investigate learning-dependent auditory attentional capture in isolation.…”