Modulation transfer functions for audiovisual speech

Abstract

Temporal synchrony between facial motion and acoustic modulations is a hallmark feature of audiovisual speech. The moving face and mouth during natural speech is known to be correlated with low-frequency acoustic envelope fluctuations (below 10 Hz), but the precise rates at which envelope information is synchronized with motion in different parts of the face are less clear. Here, we used regularized canonical correlation analysis (rCCA) to learn speech envelope filters whose outputs correlate with motion in different parts of the speakers face. We leveraged recent advances in video-based 3D facial landmark estimation allowing us to examine statistical envelope-face correlations across a large number of speakers (∼4000). Specifically, rCCA was used to learn modulation transfer functions (MTFs) for the speech envelope that significantly predict correlation with facial motion across different speakers. The AV analysis revealed bandpass speech envelope filters at distinct temporal scales. A first set of MTFs showed peaks around 3-4 Hz and were correlated with mouth movements. A second set of MTFs captured envelope fluctuations in the 1-2 Hz range correlated with more global face and head motion. These two distinctive timescales emerged only as a property of natural AV speech statistics across many speakers. A similar analysis of fewer speakers performing a controlled speech task highlighted only the well-known temporal modulations around 4 Hz correlated with orofacial motion. The different bandpass ranges of AV correlation align notably with the average rates at which syllables (3-4 Hz) and phrases (1-2 Hz) are produced in natural speech. Whereas periodicities at the syllable rate are evident in the envelope spectrum of the speech signal itself, slower 1-2 Hz regularities thus only become prominent when considering crossmodal signal statistics. This may indicate a motor origin of temporal regularities at the timescales of syllables and phrases in natural speech.

Gesture–speech coupling in persons with aphasia: A kinematic-acoustic analysis.

“…Under such more or less improved measurement conditions a whole host of avenues for further research opens up. This would not only allow for a more fine-grained analysis of gesture–speech coupling (Alviar et al, 2020; Krivokapic et al, 2017; Pedersen et al, 2022; Wagner et al, 2014). It could also expand on research by relating to studies showing that gesture kinematics change depending on the communicative context (Trujillo et al, 2018) and psychological background (Trujillo, Özyürek, et al, 2021), and that such modulation of kinematics can affect smooth conversational turn-taking (Trujillo, Levinson, & Holler, 2021).…”

Real-time control of a hearing instrument with EEG-based attention decoding

“…The system could potentially fail to generalize to this mismatching stimulus condition. Alternatively, the visual face may be correlated with audio envelope information (O’Sullivan et al, 2017a; Pedersen et al, 2022), and it may be easier to focus auditory attention on a speaker that can be seen, which may in turn improve decoding (O’Sullivan et al, 2013). Potential visual benefits are important to investigate, for instance in the case were the user wants to switch attention to a previously ignored speaker.…”

A Visual Speech Intelligibility Benefit Based on Speech Rhythm

“…From this point of view, the observed motion of the articulators (visual speech) provides a direct source of information about speech rhythm. Indeed, clearly observable peri-oral movements are significantly correlated with fluctuations in the speech envelope at rhythm rates that peak, roughly, at the syllable rate of 3 to 4 Hz [ 19 ]. Thus, for current purposes we propose that speech-rhythm information is conveyed by the motion of jaw opening and closing.…”