Interaction of bottom-up and top-down neural mechanisms in spatial multi-talker speech perception

Patel, Prachi; Heijden, Kiki van der; Bickel, Stephan; Herrero, Jose L.; Mehta, Ashesh D.; Mesgarani, Nima

doi:10.1016/j.cub.2022.07.047

Cited by 13 publications

(18 citation statements)

References 77 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, previous research has suggested that multisensory integration can happen automatically and independent of attention (i.e., preattentively) based on shared temporal properties (Van der Burg et al, 2011;Atilgan et al, 2018). Whereas we might expect a greater influence of attention on multisensory integration at longer latencies given previous research on cocktail party attention (Power et al, 2012;Patel et al, 2022).…”

Section: Single-lag Analysismentioning

confidence: 82%

“…This interpretation aligns with other findings presented in the paper. With the single-lag analysis, we identified three major time pointsconsistent with early visual and auditory responses (Schroeder et al, 2008;Karthik et al, 2022), and attentional effects (Power et al, 2012;Patel et al, 2022) -where gaze was associated with stronger speech tracking driven by non-linear multisensory effects. Intriguingly, in a much earlier epoch (−438 ms onward), speech tracking but not multisensory interactions was affected by gaze.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario

Ahmed,

Nidiffer,

Lalor

2023

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Seeing the speaker’s face greatly improves our speech comprehension in noisy environments. This is due to the brain’s ability to combine the auditory and the visual information around us, a process known as multisensory integration. Selective attention also strongly influences what we comprehend in scenarios with multiple speakers–an effect known as the cocktail-party phenomenon. However, the interaction between attention and multisensory integration is not fully understood, especially when it comes to natural, continuous speech. In a recent electroencephalography (EEG) study, we explored this issue and showed that multisensory integration is enhanced when an audiovisual speaker is attended compared to when that speaker is unattended. Here, we extend that work to investigate how this interaction varies depending on a person’s gaze behavior, which affects the quality of the visual information they have access to. To do so, we recorded EEG from 31 healthy adults as they performed selective attention tasks in several paradigms involving two concurrently presented audiovisual speakers. We then modeled how the recorded EEG related to the audio speech (envelope) of the presented speakers. Crucially, we compared two classes of model – one that assumed underlying multisensory integration (AV) versus another that assumed two independent unisensory audio and visual processes (A+V). This comparison revealed evidence of strong attentional effects on multisensory integration when participants were looking directly at the face of an audiovisual speaker. This effect was not apparent when the speaker’s face was in the peripheral vision of the participants. Overall, our findings suggest a strong influence of attention on multisensory integration when high fidelity visual (articulatory) speech information is available. More generally, this suggests that the interplay between attention and multisensory integration during natural audiovisual speech is dynamic and is adaptable based on the specific task and environment.

show abstract

Section: Single-lag Analysismentioning

confidence: 82%

Section: Discussionmentioning

confidence: 99%

The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario

Ahmed,

Nidiffer,

Lalor

2023

Front. Hum. Neurosci.

View full text Add to dashboard Cite

show abstract

“…At random moments in between trials, participants were asked to report the location of the attended talker and the last sentence uttered by the attended talker. Participants successfully performed the behavioral task (see 24 for a detailed analysis of the behavioral results).…”

Section: Resultsmentioning

confidence: 99%

Joint population coding and temporal coherence link an attended talker’s voice and location features in naturalistic multi-talker scenes

van der Heijden,

Patel,

Bickel

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Listeners readily extract multi–dimensional auditory objects such as a ´localized talker´ from complex acoustic scenes with multiple talkers. Yet, the neural mechanisms underlying simultaneous encoding and linking of different sound features — for example, a talker´s voice and location — are poorly understood. We analyzed invasive intracranial recordings in neurosurgical patients attending to a localized talker in real–life cocktail party scenarios. We found that sensitivity to an individual talker´s voice and location features was distributed throughout auditory cortex and that neural sites exhibited a gradient from sensitivity to a single feature to joint sensitivity to both features. On a population level, cortical response patterns of both dual–feature sensitive sites but also single–feature sensitive sites revealed simultaneous encoding of an attended talker´s voice and location features. However, for single–feature sensitive sites, the representation of the primary feature was more precise. Further, sites which selective tracked an attended speech stream concurrently encoded an attended talker´s voice and location features, indicating that such sites combine selective tracking of an attended auditory object with encoding of the object´s features. Finally, we found that attending a localized talker selectively enhanced temporal coherence between single–feature voice sensitive sites and single–feature location sensitive sites, providing an additional mechanism for linking voice and location features in multi–talker scenes. These results demonstrate that a talker´s voice and location features are linked during multi-dimensional object formation in naturalistic multi–talker scenes by joint population coding as well as by temporal coherence between neural sites.

show abstract

“…This model not only separates the speech of moving talkers but also allows listeners to accurately track their locations, an aspect crucial for realistic AAD applications. An essential contribution of our study is that incorporating real-time talker trajectories estimated by the speech separation algorithm in addition to spectro-temporal information can improve AAD accuracy 19,[35][36][37] . Further research is needed to distinguish listener motion-induced from talker motion-induced acoustic change and how it could be encoded differently in the human auditory cortex 38,39 .…”

Section: Discussionmentioning

confidence: 99%

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

Choudhari,

Han,

Bickel

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Focusing on a specific conversation amidst multiple interfering talkers presents a significant challenge, especially for the hearing-impaired. Brain-controlled assistive hearing devices aim to alleviate this problem by separating complex auditory scenes into distinct speech streams and enhancing the attended speech based on the listener’s neural signals using auditory attention decoding (AAD). Departing from conventional AAD studies that relied on oversimplified scenarios with stationary talkers, we present a realistic AAD task that mirrors the dynamic nature of acoustic settings. This task involves focusing on one of two concurrent conversations, with multiple talkers taking turns and moving continuously in space with background noise. Invasive electroencephalography (iEEG) data were collected from three neurosurgical patients as they focused on one of the two moving conversations. We propose an enhanced brain-controlled assistive hearing system that combines AAD and a binaural speaker-independent speech separation model. The separation model unmixes talkers while preserving their spatial location and provides talker trajectories to the neural decoder to improve auditory attention decoding accuracy. Our subjective and objective evaluations show that the proposed system enhances speech intelligibility and facilitates conversation tracking while maintaining spatial cues and voice quality in challenging acoustic environments. This research demonstrates the potential of our approach in real-world scenarios and marks a significant step towards developing assistive hearing technologies that adapt to the intricate dynamics of everyday auditory experiences.TAKEAWAYS-Brain-controlled hearing device for scenarios with moving conversations in multi-talker settings, closely mimicking real-world listening environments-Developed a binaural speech separation model that separates speech of moving talkers while retaining their spatial locations, enhancing auditory perception and auditory attention decoding-Proposed system enhances speech intelligibility and reduces listening effort in realistic acoustic scenes

show abstract

Interaction of bottom-up and top-down neural mechanisms in spatial multi-talker speech perception

Cited by 13 publications

References 77 publications

The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario

The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario

Joint population coding and temporal coherence link an attended talker’s voice and location features in naturalistic multi-talker scenes

Brain-controlled augmented hearing for spatially moving conversations in multi-talker environments

Contact Info

Product

Resources

About