Speaker-Adaptive Acoustic-Articulatory Inversion Using Cascaded Gaussian Mixture Regression

Abstract

This article addresses the adaptation of an acoustic-articulatory model of a reference speaker to the voice of another speaker, using a limited amount of audio-only data. In the context of pronunciation training, a virtual talking head displaying the internal speech articulators (e.g. the tongue) could be automatically animated by means of such a model using only the speaker's voice. In this study, the articulatory-acoustic relationship of the reference speaker is modeled by a gaussian mixture model (GMM). To address the speaker adaptation problem, we propose a new framework called cascaded Gaussian mixture regression (C-GMR), and derive two implementations. The first one, referred to as Split-C-GMR, is a straightforward chaining of two distinct GMRs: one mapping the acoustic features of the source speaker into the acoustic space of the reference speaker, and the other estimating the articulatory trajectories with the reference model. In the second implementation, referred to as Integrated-C-GMR, the two mapping steps are tied together in a single probabilistic model. For this latter model, we present the full derivation of the exact EM training algorithm, that explicitly exploits the missing data methodology of machine learning. Other adaptation schemes based on maximum-a-posteriori (MAP), maximum likelihood linear regression (MLLR) and direct crossspeaker acoustic-to-articulatory GMR are also investigated. Experiments conducted on two speakers for different amount of adaptation data show the interest of the proposed C-GMR techniques.

Adaptation of a Gaussian Mixture Regressor to a New Input Distribution: Extending the C-GMR Framework

“…In this section, we briefly recall three approaches for GMR adaptation considered in [11], which will be used here as a baseline. Their graphical representation is illustrated in Fig.…”

“…Similarly to [11], the initialization of the proposed EM algorithm takes a very peculiar aspect. Indeed, the reference (X, Y) GMR model is used to initialize the marginal parameters in (X, Y) of the Joint GMM.…”

“…However, the EM for the IC-GMR presented in [11] (which exploits the linear-Gaussian form of the IC-GMR) is not derivable as a particular case of the EM of Section 4. More precisely, if one attempts to estimate the IC-GMR parameters with the algorithm we present in this paper, the M-step should be constrained by (5).…”

“…For example, in speech processing, this happens when considering a speaker different from the one used for training. To address this problem, we first proposed in [8] to adapt the model parameters related to input observations using two state-of-the-art adaptation techniques for GMM which are maximum a posteriori (MAP) [9] and maximum likelihood linear regression (MLLR) [10]. Then, we proposed in [11] a general framework called Cascaded GMR (C-GMR) and derived two implementations (see Fig.…”

“…The IC-GMR put into practice the general missing data methodology of machine learning [12] to exploit both the small and potentially sparse adaptation dataset and the large and dense training dataset. In [11], this model was shown to provide superior performance to the SC-GMR and to a s a direct GMR (D-GMR) trained only with new input data and corresponding output data. The D-GMR, SC-GMR and IC-GMR are briefly presented in Section 2.…”