Masking of the mouth area impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Haider, Chandra Leon; Suess, Nina; Hauswald, Anne; Park, Hyojin; Weisz, Nathan

doi:10.1016/j.neuroimage.2022.119044

Cited by 22 publications

(41 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We next implemented a receptive field model, also known as a multivariate temporal response function (mTRF) (Crosse et al, 2016), which can be interpreted as a linear filter in the brain processing a stimulus feature (speech envelope S(t)) mapped onto the continuous neural responses (MEG response over time, r(t)). The approach has been used in recent studies to study the mapping between brain responses and naturalistic speech feature representations (Broderick et al, 2018; Teng et al, 2021; Haider et al, 2022). The schematic flowchart for our analysis is shown in Fig.…”

Section: Resultsmentioning

confidence: 99%

“…An increasing number of studies has demonstrated that neural representations of acoustic (Park et al, 2015; Park et al, 2016; Hauswald et al, 2018; Park et al, 2018b; Park et al, 2018a; Biau et al, 2021; Haider et al, 2022) and linguistic, e.g., semantic features (Kutas and Federmeier, 2011; Strauss et al, 2014; Huth et al, 2016; Wang et al, 2018; Broderick et al, 2019; Kaufeld et al, 2020) of naturalistic auditory or audiovisual speech are quantifiable in Magnetoencephalography (MEG) or Electroencephalography (EEG) recordings based on frequency-domain synchronization analysis or time-domain regression analysis. Furthermore, recent developments of natural language processing (NLP) models based on machine learning algorithms, such as word vectors (Mikolov et al, 2013), have brought breakthroughs not only to the area of artificial intelligence (AI), for example, speech/text recognition, machine translations (e.g., speech-to-text), but also to the neuroscientific study of rich, naturalistic speech stimuli (Broderick et al, 2018; Pereira et al, 2018) or movie (Nishida et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Get the gist of the story: Neural map of topic keywords in multi-speaker environment

Park

Groß

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Neural representation of lexico-semantics in speech processing has been revealed in recent years. However, to date, how the brain makes sense of the higher-level semantic gist (topic keywords) of a continuous speech remains mysterious. Capitalizing on a generative probabilistic topic modelling algorithm on speech materials to which participants listened while their brain activities were recorded by Magnetoencephalography (MEG), here we show spatio-temporal neural representation of topic keywords in a multi-speaker environment where task-relevant (attended) and -irrelevant (unattended) speech co-exits. We report the difference of neural representation between salient and less salient semantic gist of both attended and unattended speech. Moreover, we show that greater sensitivity to semantically salient unattended speech in the left auditory and motor cortices negatively mediates attended speech comprehension.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Get the gist of the story: Neural map of topic keywords in multi-speaker environment

Park

Groß

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Adult studies also suggest that face masks do not alter all aspects of speech processing. For instance, the following aspects have been shown to be negatively impacted: accuracy of word/sentence translation (Rahne et al, 2021 ), accuracy of speech recognition (e.g., Magee et al, 2020 ; Yi et al, 2021 ), listening effort (Haider et al, 2022 ), especially for hearing-impaired populations (Lee et al, 2022 ), intelligibility in noisy environments (Brown et al, 2021 ), or cortical tracking/reconstruction of audiovisual speech, especially reconstruction of high-level segmental features (i.e., phoneme and the word onset) in more challenging listening conditions (Haider et al, 2022 ). Interestingly, if speech is not placed in a noisy/challenging environment, intelligibility is usually intact (Magee et al, 2020 ; Brown et al, 2021 ; Rahne et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

Early Word Segmentation Behind the Mask

et al. 2022

View full text Add to dashboard Cite

Infants have been shown to rely both on auditory and visual cues when processing speech. We investigated the impact of COVID-related changes, in particular of face masks, in early word segmentation abilities. Following up on our previous study demonstrating that, by 4 months, infants already segmented targets presented auditorily at utterance-edge position, and, using the same visual familiarization paradigm, 7–9-month-old infants performed an auditory and an audiovisual word segmentation experiment in two conditions: without and with an FFP2 face mask. Analysis of acoustic and visual cues showed changes in face-masked speech affecting the amount, weight, and location of cues. Utterance-edge position displayed more salient cues than utterance-medial position, but the cues were attenuated in face-masked speech. Results revealed no evidence for segmentation, not even at edge position, regardless of mask condition and auditory or visual speech presentation. However, in the audiovisual experiment, infants attended more to the screen during the test trials when familiarized with without mask speech. Also, the infants attended more to the mouth and less to the eyes in without mask than with mask. In addition, evidence for an advantage of the utterance-edge position in emerging segmentation abilities was found. Thus, audiovisual information provided some support to developing word segmentation. We compared 7–9-monthers segmentation ability observed in the Butler and Frota pre-COVID study with the current auditory without mask data. Mean looking time for edge was significantly higher than unfamiliar in the pre-COVID study only. Measures of cognitive and language development obtained with the CSBS scales showed that the infants of the current study scored significantly lower than the same-age infants from the CSBS (pre-COVID) normative data. Our results suggest an overall effect of the pandemic on early segmentation abilities and language development, calling for longitudinal studies to determine how development proceeds.

show abstract

“…The little research that came prior to the pandemic had found no added cost in speech perception, for normal hearing adults listening to speech through a mask ( Mendel et al, 2008 ; Atcherson et al, 2017 ). But several recent papers have reported decreases in speech perception in mask contexts ( Radonovich et al, 2009 ; Yi et al, 2021 ), due to their impact on auditory ( Rahne et al, 2021 ), or visual cues ( Haider et al, 2022 ). Interestingly, some of those new laboratory studies presented more ecological conditions (e.g., conversational speech stimuli or the use of human speech as noise).…”

Section: Introductionmentioning

confidence: 99%

Mask wearing in Japanese and French nursery schools: The perceived impact of masks on communication

et al. 2022

View full text Add to dashboard Cite

Due to the global COVID-19 pandemic, covering the mouth region with a face mask became pervasive in many regions of the world, potentially impacting how people communicate with and around children. To explore the characteristics of this masked communication, we asked nursery school educators, who have been at the forefront of daily masked interaction with children, about their perception of daily communicative interactions while wearing a mask in an online survey. We collected data from French and Japanese nursery school educators to gain an understanding of commonalities and differences in communicative behavior with face masks given documented cultural differences in pre-pandemic mask wearing habits, face scanning patterns, and communicative behavior. Participants (177 French and 138 Japanese educators) reported a perceived change in their own communicative behavior while wearing a mask, with decreases in language quantity and increases in language quality and non-verbal cues. Comparable changes in their team members’ and children’s communicative behaviors were also reported. Moreover, our results suggest that these changes in educators’ communicative behaviors are linked to their attitudes toward mask wearing and their potential difficulty in communicating following its use. These findings shed light on the impact of pandemic-induced mask wearing on children’s daily communicative environment.

show abstract

Masking of the mouth area impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Cited by 22 publications

References 58 publications

Get the gist of the story: Neural map of topic keywords in multi-speaker environment

Get the gist of the story: Neural map of topic keywords in multi-speaker environment

Early Word Segmentation Behind the Mask

Mask wearing in Japanese and French nursery schools: The perceived impact of masks on communication

Contact Info

Product

Resources

About