Putative mechanisms mediating tolerance for audiovisual stimulus onset asynchrony

Abstract

Bhat J, Miller LM, Pitt MA, Shahin AJ. Putative mechanisms mediating tolerance for audiovisual stimulus onset asynchrony. J Neurophysiol 113: [1437][1438][1439][1440][1441][1442][1443][1444][1445][1446][1447][1448][1449][1450] 2015. First published December 10, 2014; doi:10.1152/jn.00200.2014.-Audiovisual (AV) speech perception is robust to temporal asynchronies between visual and auditory stimuli. We investigated the neural mechanisms that facilitate tolerance for audiovisual stimulus onset asynchrony (AVOA) with EEG. Individuals were presented with AV words that were asynchronous in onsets of voice and mouth movement and judged whether they were synchronous or not. Behaviorally, individuals tolerated (perceived as synchronous) longer AVOAs when mouth movement preceded the speech (V-A) stimuli than when the speech preceded mouth movement (A-V). Neurophysiologically, the P1-N1-P2 auditory evoked potentials (AEPs), time-locked to sound onsets and known to arise in and surrounding the primary auditory cortex (PAC), were smaller for the in-sync than the out-of-sync percepts. Spectral power of oscillatory activity in the beta band (14 -30 Hz) following the AEPs was larger during the in-sync than out-of-sync perception for both A-V and V-A conditions. However, alpha power (8 -14 Hz), also following AEPs, was larger for the in-sync than out-of-sync percepts only in the V-A condition. These results demonstrate that AVOA tolerance is enhanced by inhibiting low-level auditory activity (e.g., AEPs representing generators in and surrounding PAC) that code for acoustic onsets. By reducing sensitivity to acoustic onsets, visual-to-auditory onset mapping is weakened, allowing for greater AVOA tolerance. In contrast, beta and alpha results suggest the involvement of higherlevel neural processes that may code for language cues (phonetic, lexical), selective attention, and binding of AV percepts, allowing for wider neural windows of temporal integration, i.e., greater AVOA tolerance.audiovisual stimulus onset asynchrony; auditory evoked potentials; alpha band activity; beta band activity; EEG; oscillatory activity SPEECH IS THE MOST IMPORTANT communication tool we have.

Tolerance for audiovisual asynchrony is enhanced by the spectrotemporal fidelity of the speaker’s mouth movements and speech

“…Additionally, it is related to the place and manner of articulation of consonants and the salience of tongue and lip dynamics (front/back tongue, round/unrounded lips) of vowels (Vatakis et al, 2012). One consistent finding is that individuals tolerate greater AVOA when visual events lead auditory events as opposed to the reverse (Bhat et al, 2015; Conrey & Pisoni, 2006; Eg, Griwodz, Halvorsen, & Behne, 2015; Magnotti, Ma, & Beauchamp, 2013; Miller & D’Esposito, 2005; Stevenson et al, 2013), although this effect tend to be less asymmetrical for speech stimuli (Stevenson et al, 2013). Moreover, this asymmetry effect is not observed in single-modality scenarios (auditory-auditory).…”

“…The metric of AV integration we examine here is tolerance for audiovisual onset asynchrony (AVOA) in spoken language (Bhat, Miller, Pitt, & Shahin, 2015). In Bhat et al (2015), AVOA tolerance was quantified by measuring individual thresholds of maximum asynchrony at which temporally misaligned mouth movements and corresponding speech sounds were perceived as synchronous.…”

“…The metric of AV integration we examine here is tolerance for audiovisual onset asynchrony (AVOA) in spoken language (Bhat, Miller, Pitt, & Shahin, 2015). In Bhat et al (2015), AVOA tolerance was quantified by measuring individual thresholds of maximum asynchrony at which temporally misaligned mouth movements and corresponding speech sounds were perceived as synchronous. AVOA tolerance can be assessed when visual speech leads acoustic speech, or vice versa, and is rooted in the unity assumption that individuals perceive signals from two sensory modalities that share several qualities as one multisensory event (Vatakis, Ghazanfar, & Spence, 2008; Vatakis & Spence, 2007; Welch & Warren, 1980).…”

“…Bhat et al, (2015) proposed a neural model (termed the Dynamic Reweighting Model) for how the AV integration window behaves as a function of the spectrotemporal fidelity of audiovisual stimuli. They posited that the spectrotemporal information of AV stimuli could modulate the neural pathways for processing the speech signal and in turn cause the AV window of integration to resize.…”

“…Concurrent with this upward shift is the weakening of activity at low-level networks in the primary auditory cortex, which encodes simple acoustic features such as acoustic onsets (Shahin et al, 2009). This Dynamic Reweighting Model was further motivated by the results of Bhat et al (2015) showing reduced auditory evoked potentials to onsets of words that were perceived as synchronous as opposed to asynchronous with the corresponding mouth movements. A key aspect of this reweighting is that it also modifies the connections between the auditory and visual systems.…”