Differential effects of intra-modal and cross-modal reward value on visual perception: ERP evidence

Abstract

Stimuli associated with high reward modulate perception and such value-driven effects have been shown to originate from the modulation of the earliest stages of sensory processing in the brain. In natural environments objects comprise multiple features (imagine a rolling soccer ball, with its black and white patches and the swishing sound made during its motion), where each feature may signal different associations with previously encountered rewards. How perception of such an object is affected by the value associations of its constituent parts is unknown. The present study compares intra- and cross-modal value-driven effects on behavioral and electrophysiological correlates of visual perception. Human participants first learned the reward associations of visual and auditory cues. Subsequently, they performed a visual orientation discrimination task in the presence of previously rewarded visual or auditory cues (intra- and cross-modal cues, respectively) that were concurrently presented with the target stimulus. During the conditioning phase, when reward associations were learned and reward cues were the target of the task, reward value of both modalities enhanced the electrophysiological correlates of sensory processing in visual cortex. During the post-conditioning phase, when reward delivery was halted and previously rewarded stimuli were task-irrelevant, cross-modal value-enhanced behavioral measures of visual sensitivity whereas intra-modal value led to a trend for suppression. A similar pattern of modulations was found in the simultaneously recorded event-related potentials (ERPs) of posterior electrodes. We found an early (90-120 ms) suppression of ERPs evoked by high-value, intra-modal stimuli. Cross-modal cues led to a later value-driven modulation, with an enhancement of response positivity for high- compared to low-value stimuli starting at the N1 window (180-250 ms) and extending to the P3 (300-600 ms) responses of the posterior electrodes. These results indicate that visual cortex is modulated by the reward value of visual as well as auditory cues. Previously rewarded, task-irrelevant cues from the same or different sensory modality have a different effect on visual perception, as intra-modal high-value cues may interfere with the target processing, whereas cross-modal high-value cues boost the perception of the target.

Value-driven modulation of visual perception by visual and auditory reward cues: The role of performance-contingent delivery of reward

“…However, we note that PR phase in our experiment was only tested after the PC phase, and therefore participants had a long exposure to the reward cues in a setting when they were predictive of the reward delivery when orientation discrimination task was performed correctly (i.e., the PC phase). In contrast, in our previous work (Vakhrushev et al, 2021), we tested the PR phase only after a conditioning phase which employed a different task (i.e., cue localization) than the test phase (i.e., orientation discrimination), and found that reward effects were most prominent for cross-modal cues. Together, the current results and results of our previous study indicate that the effects of reward critically depend on the training mode (Jahfari and Theeuwes, 2017;Failing and Theeuwes, 2018) and the relation between the rewarded stimuli and the task-relevant features.…”

“…In total, 43 subjects participated in the experiment to fulfill a target sample size of N = 36 based on a previous study (Vakhrushev et al, 2021). They were invited via an online recruiting system.…”

“…identical to a previous study (Vakhrushev et al, 2021). The paradigm employed during the conditioning was a spatial localization task (see Supplementary Figure 1 and the Section "Experimental procedure") where participants reported the side (left or right) from which visual or auditory stimuli were presented.…”

“…Using this design, it was shown that task-irrelevant auditory cues that were previously associated with high reward enhanced the visual sensitivity compared to low reward cues. A followup study (Vakhrushev et al, 2021) used a similar design and compared task-irrelevant reward cues from the same (vision) or different (audition) sensory modality in terms of their effect on perceptual decisions made about a visual target. In this study, it was found that PR auditory and visual cues had distinct effects on behavioral and electrophysiological correlates of visual perception, suggesting that reward-driven modulations may have dependencies on the sensory modality of task-irrelevant stimuli.…”

“…Therefore, in the current study, we designed a paradigm that tested the effect of three factors on visual perception: reward magnitude, sensory modalities of reward cues, and the contingency of reward delivery on task performance. Specifically, a similar design as two previous studies from our lab (Pooresmaeili et al, 2014;Vakhrushev et al, 2021) was used where auditory or visual cues were first associated with either high or low monetary reward during a training phase (referred to as conditioning). During the test phase, auditory and visual cues were presented at the same time as the target of a visual discrimination task but did not carry any information about the task at hand (i.e., orientation discrimination).…”