Occlusion of lip movements impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Abstract

Face masks have become a prevalent measure during the Covid-19 pandemic to counteract the transmission of SARS-CoV 2. An unintended 'side-effect' of face masks is their adverse influence on speech perception especially in challenging listening situations. So far, behavioural studies have not pinpointed exactly which feature(s) of speech processing face masks affect in such listening situations. We conducted an audiovisual (AV) multi-speaker experiment using naturalistic speech (i.e. an audiobook). In half of the trials, the target speaker wore a (surgical) face mask, while we measured the brain activity of normal hearing participants via magnetoencephalography (MEG). A decoding model on the clear AV speech (i.e. no additional speaker and target speaker not wearing a face mask) was trained and used to reconstruct crucial speech features in each condition. We found significant main effects of face masks on the reconstruction of acoustic features, such as the speech envelope and spectral speech features (i.e. pitch and formant frequencies), while reconstruction of higher level features of speech segmentation (phoneme and word onsets) were especially impaired through masks in difficult listening situations, i.e. when a distracting speaker was also presented. Our findings demonstrate the detrimental impact face masks have on listening and speech perception, thus extending previous behavioural results. Supporting the idea of visual facilitation of speech is the fact that we used surgical face masks in our study, which only show mild effects on speech acoustics. This idea is in line with recent research, also by our group, showing that visual cortical regions track spectral modulations. Since hearing impairment usually affects higher frequencies, the detrimental effect of face masks might pose a particular challenge for individuals who likely need the visual information about higher frequencies (e.g. formants) to compensate.

The use of lexical semantics for processing face-masked speech in monolinguals and bilinguals

“…Face masks cover visual articulators which are important for face-to-face communication (Jesse and Massaro, 2010;Campbell, 2008). Haider et al (2023) demonstrate, for example, that it is the inhibition of visual speech cues caused by face masks, and not the inhibition of acoustic information, that affects speech processing. Still, by impeding both of these signals-which are crucial for accurate perception-mask wearing impacts the receiver's ability to comprehend the utterance (Brown et al, 2021;Fort et al, 2013) and recognize words (Truong et al, 2021).…”