Audiovisual speech perception has been frequently studied considering phoneme, syllable and word processing levels. Here, we examined the constraints that visual speech information might exert during the recognition of words embedded in a natural sentence context. We recorded event-related potentials (ERPs) to words that could be either strongly or weakly predictable on the basis of the prior semantic sentential context and, whose initial phoneme varied in the degree of visual saliency from lip movements. When the sentences were presented audio-visually (Experiment 1), words weakly predicted from semantic context elicited a larger long-lasting N400, compared to strongly predictable words. This semantic effect interacted with the degree of visual saliency over a late part of the N400. When comparing audio-visual versus auditory alone presentation (Experiment 2), the typical amplitude-reduction effect over the auditory-evoked N100 response was observed in the audiovisual modality. Interestingly, a specific benefit of high- versus low-visual saliency constraints occurred over the early N100 response and at the late N400 time window, confirming the result of Experiment 1. Taken together, our results indicate that the saliency of visual speech can exert an influence over both auditory processing and word recognition at relatively late stages, and thus suggest strong interactivity between audio-visual integration and other (arguably higher) stages of information processing during natural speech comprehension.

Temporal recalibration of cross-modal synchrony has been proposed as a mechanism to compensate for timing differences between sensory modalities. However, far from the rich complexity of everyday life sensory environments, most studies to date have examined recalibration on isolated cross-modal pairings. Here, we hypothesize that selective attention might provide an effective filter to help resolve which stimuli are selected when multiple events compete for recalibration. We addressed this question by testing audio-visual recalibration following an adaptation phase where two opposing audio-visual asynchronies were present. The direction of voluntary visual attention, and therefore to one of the two possible asynchronies (flash leading or flash lagging), was manipulated using colour as a selection criterion. We found a shift in the point of subjective audio-visual simultaneity as a function of whether the observer had focused attention to audio-then-flash or to flash-then-audio groupings during the adaptation phase. A baseline adaptation condition revealed that this effect of endogenous attention was only effective toward the lagging flash. This hints at the role of exogenous capture and/or additional endogenous effects producing an asymmetry toward the leading flash. We conclude that selective attention helps promote selected audio-visual pairings to be combined and subsequently adjusted in time but, stimulus organization exerts a strong impact on recalibration. We tentatively hypothesize that the resolution of recalibration in complex scenarios involves the orchestration of top-down selection mechanisms and stimulus-driven processes.

The interactions between the senses are essential for cognitive functions such as perception, attention, and action planning. Past research helped understanding of multisensory processes in the laboratory. Yet, the efforts to extrapolate these findings to the real-world are scarce. Extrapolation to real-world contexts is important for practical and theoretical reasons. Multisensory phenomena might be expressed differently in real-world settings compared to simpler laboratory situations. Some effects might become stronger, others may disappear, and new outcomes could be discovered. This Element discusses research that uncovers multisensory interactions under complex environments, with an emphasis on the interplay of multisensory mechanisms with other processes.

In everyday life multisensory events, such as a glass crashing on the floor, the different sensory inputs are often experienced as simultaneous, despite the sensory processing of sound and sight within the brain are temporally misaligned. This lack of cross‐modal synchrony is the unavoidable consequence of different light and sound speeds, and their different neural transmission times in the corresponding sensory pathways. Hence, cross‐modal synchrony must be reconstructed during perception. It has been suggested that spontaneous fluctuations in neural excitability might be involved in the temporal organisation of sensory events during perception and account for variability in behavioural performance. Here, we addressed the relationship between ongoing brain oscillations and the perception of cross‐modal simultaneity. Participants performed an audio‐visual simultaneity judgement task while their EEG was recorded. We focused on pre‐stimulu activity, and found that the phase of neural oscillations at 13 ± 2 Hz 200 ms prior to the stimulus correlated with subjective simultaneity of otherwise identical sound‐flash events. Remarkably, the correlation between EEG phase and behavioural report occurred in the absence of concomitant changes in EEG amplitude. The probability of simultaneity perception fluctuated significantly as a function of pre‐stimulus phase, with the largest perceptual variation being accounted for phase angles nearly 180º apart. This pattern was strongly reliable for sound‐flash pairs but not for flash‐sound pairs. Overall, these findings suggest that the phase of ongoing brain activity might underlie internal states of the observer that influence cross‐modal temporal organisation between the senses and, in turn, subjective synchrony.

Background Past studies do not account for avoidance behaviour in migraine as a potential confounder of phonophobia. Objective To analyse whether phonophobia is partially driven by avoidance behaviour when using the classic methodology (method of limits). Methods This is a case-control study where we tested phonophobia in a cohort of high-frequency/chronic migraine patients (15.5 ± 0.74 headache days/month) and non-headache controls. Auditory stimuli, delivered in both ears, were presented using three different paradigms: the method of limits, the method of constant stimuli, and the adaptive method. Participants were asked to report how bothersome each tone was until a sound aversion threshold was estimated for each method. Results In this study, we successfully replicate previously reported reduction in sound aversion threshold using three different methods in a group of 35 patients and 25 controls (p < 0.0001). Avoidance behaviour in migraine reduced sound aversion threshold in the method of limits (p = 0.0002) and the adaptive method (p < 0.0001) when compared to the method of constant stimuli. While thresholds in controls remained the same across methods (method of limits, p = 0.9877 and adaptive method, p = 1). Conclusion Avoidance behaviour can exacerbate phonophobia. The current methodology to measure phonophobia needs to be revised.