Using deep neural networks to estimate tongue movements from speech face motion

Abstract

This study concludes a tripartite investigation into the indirect visibility of the moving tongue in human speech as reflected in co-occurring changes of the facial surface. We were in particular interested in how the shared information is distributed over the range of contributing frequencies. In the current study we examine the degree to which tongue movements during speech can be reliably estimated from face motion using artificial neural networks. We simultaneously acquired data for both movement types; tongue movements were measured with Electromagnetic Articulography (EMA), face motion with a passive marker-based motion capture system. A multiresolution analysis using wavelets provided the desired decomposition into frequency subbands. In the two earlier studies of the project we established linear and non-linear relations between lingual and facial speech motions, as predicted and compatible with previous research in auditory-visual speech. The results of the current study using a Deep Neural Network (DNN) for prediction show that a substantive amount of variance can be recovered (between 13.9 and 33.2% dependent on the speaker and tongue sensor location). Importantly, however, the recovered variance values and the root mean squared error values of the Euclidean distances between the measured and the predicted tongue trajectories are in the range of the linear estimations of our earlier study.

Improving Ultrasound Tongue Image Reconstruction from Lip Images Using Self-supervised Learning and Attention Mechanism

“…As aforementioned, these imaging modalities are incontrovertibly statistically related each other. In [21], the authors explored the use of deep neural networks to estimate the tongue's motion from the face pictures. In this paper, we follow the task defined in [39] and we aim to picture tongue's motion from the lip images, leveraging the ultrasound tongue imaging.…”

“…These imaging modalities are incontrovertibly statistically related each other, such as recent studies suggest that one can picture the corresponding tongue motion from their voice and vice versa [26]. Here, we would like to ask a related question: given an observable image sequences of lips, can we predict the motion of the tongue [21]. The authors in [39] demonstrated that deep learning model [12,20] can reconstruct the tongue's motion from the lip images with satisfactory performance.…”