Predicting tongue positions from acoustics and facial features

Toutios, Asterios; Ouni, Slim

doi:10.21437/interspeech.2011-680

Cited by 3 publications

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

The use of articulatory movement data in speech synthesis applications: An overview — Application of articulatory movements using machine learning algorithms —

Richmond

Ling

Yamagishi

2015

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

This paper describes speech processing work in which articulator movements are used in conjunction with the acoustic speech signal and/or linguistic information. By ''articulator movements,'' we mean the changing positions of human speech articulators such as the tongue and lips, which may be recorded by electromagnetic articulography (EMA), amongst other articulography techniques. Specifically, we provide an overview of: i) inversion mapping techniques, where we estimate articulator movements from a given new speech waveform automatically; ii) statistical voice conversion and speech synthesis techniques which use articulator movements as part of the process to generate synthetic speech, and also make it intuitively controllable via articulation; and iii) automatic prediction (or synthesis) of articulator movements from any given new text input.

show abstract

The use of articulatory movement data in speech synthesis applications: An overview — Application of articulatory movements using machine learning algorithms —

Richmond

Ling

Yamagishi

2015

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

show abstract

Using deep neural networks to estimate tongue movements from speech face motion

Kroos¹,

Bundgaard‐Nielsen²,

Best³

et al. 2017

The 14th International Conference on Auditory-Visual Speech Processing

View full text Add to dashboard Cite

This study concludes a tripartite investigation into the indirect visibility of the moving tongue in human speech as reflected in co-occurring changes of the facial surface. We were in particular interested in how the shared information is distributed over the range of contributing frequencies. In the current study we examine the degree to which tongue movements during speech can be reliably estimated from face motion using artificial neural networks. We simultaneously acquired data for both movement types; tongue movements were measured with Electromagnetic Articulography (EMA), face motion with a passive marker-based motion capture system. A multiresolution analysis using wavelets provided the desired decomposition into frequency subbands. In the two earlier studies of the project we established linear and non-linear relations between lingual and facial speech motions, as predicted and compatible with previous research in auditory-visual speech. The results of the current study using a Deep Neural Network (DNN) for prediction show that a substantive amount of variance can be recovered (between 13.9 and 33.2% dependent on the speaker and tongue sensor location). Importantly, however, the recovered variance values and the root mean squared error values of the Euclidean distances between the measured and the predicted tongue trajectories are in the range of the linear estimations of our earlier study.

show abstract

On the evaluation of inversion mapping performance in the acoustic domain

et al. 2013

View full text Add to dashboard Cite

The two measures typically used to assess the performance of an inversion mapping method, where the aim is to estimate what articulator movements gave rise to a given acoustic signal, are root mean squared (RMS) error and correlation. In this paper, we investigate whether "task-based" evaluation using an articulatory-controllable HMM-based speech synthesis system can give useful additional information to complement these measures. To assess the usefulness of this evaluation approach, we use articulator trajectories estimated by a range of different inversion mapping methods as input to the synthesiser, and measure their performance in the acoustic domain in terms of RMS error of the generated acoustic parameters and with a listening test involving 30 participants. We then compare these results with the standard RMS error and correlation measures calculated in the articulatory domain. Interestingly, in the acoustic evaluation we observe one method performs with no statistically significant difference from measured articulatory data, and cases where statistically significant differences between methods exist which are not reflected in the results of the two standard measures. From our results, we conclude such task-based evaluation can indeed provide interesting extra information, and gives a useful way to compare inversion methods.

show abstract

Predicting tongue positions from acoustics and facial features

Cited by 3 publications

References 15 publications

The use of articulatory movement data in speech synthesis applications: An overview — Application of articulatory movements using machine learning algorithms —

The use of articulatory movement data in speech synthesis applications: An overview — Application of articulatory movements using machine learning algorithms —

Using deep neural networks to estimate tongue movements from speech face motion

On the evaluation of inversion mapping performance in the acoustic domain

Contact Info

Product

Resources

About