Multimodal voice conversion based on non-negative matrix factorization

Abstract

A multimodal voice conversion (VC) method for noisy environments is proposed. In our previous non-negative matrix factorization (NMF)-based VC method, source and target exemplars are extracted from parallel training data, in which the same texts are uttered by the source and target speakers. The input source signal is then decomposed into source exemplars, noise exemplars, and their weights. Then, the converted speech is constructed from the target exemplars and the weights related to the source exemplars. In this study, we propose multimodal VC that improves the noise robustness of our NMF-based VC method. Furthermore, we introduce the combination weight between audio and visual features and formulate a new cost function to estimate audiovisual exemplars. Using the joint audiovisual features as source features, VC performance is improved compared with that of a previous audio-input exemplar-based VC method. The effectiveness of the proposed method is confirmed by comparing its effectiveness with that of a conventional audio-input NMF-based method and a Gaussian mixture model-based method.

Parallel Dictionary Learning for Multimodal Voice Conversion Using Matrix Factorization

“…The noise in the input signal is not only output with the converted signal, but may also degrade the conversion performance itself due to unexpected mapping of source features. In [16], the experimental result proved the effectiveness of visual features in the task of VC in noisy environments.…”

“…In [16], we have proposed an exemplar-based multimodal Voice Conversion (VC) method. In the task of automatic speech recognition (ASR), one problem is that the recognition performance remarkably decreases under noisy environments, and it becomes a significant problem seeking to develop a practical use of ASR.…”