Non-Parallel Voice Conversion Using Variational Autoencoders Conditioned by Phonetic Posteriorgrams and D-Vectors

Saito, Yuki; Ijima, Yusuke; Nishida, K.; Takamichi, Shinnosuke

doi:10.1109/icassp.2018.8461384

Cited by 110 publications

(71 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, the challenge in developing the non-parallel spectral conversion model has attracted many works within the recent years, such as: with the use of clustered spectral matching algorithms [12,13]; with adaptation/alignment of speaker model parameters [14,15]; with restricted Boltzmann machine [16]; with generative adversarial networks (GAN)-based methods [17,18]; and with variational autoencoder (VAE)-based frameworks [19,20,21,22]. In this work, we focus on the use of VAE-based system, due to its potential in employing latent space to represent common hidden aspects of speech signal, between different speakers, e.g., phonetical attributes.…”

Section: Introductionmentioning

confidence: 99%

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

In this paper, we present a novel technique for a non-parallel voice conversion (VC) with the use of cyclic variational autoencoder (CycleVAE)-based spectral modeling. In a variational autoencoder (VAE) framework, a latent space, usually with a Gaussian prior, is used to encode a set of input features. In a VAE-based VC, the encoded latent features are fed into a decoder, along with speaker-coding features, to generate estimated spectra with either the original speaker identity (reconstructed) or another speaker identity (converted). Due to the non-parallel modeling condition, the converted spectra can not be directly optimized, which heavily degrades the performance of a VAEbased VC. In this work, to overcome this problem, we propose to use CycleVAE-based spectral model that indirectly optimizes the conversion flow by recycling the converted features back into the system to obtain corresponding cyclic reconstructed spectra that can be directly optimized. The cyclic flow can be continued by using the cyclic reconstructed features as input for the next cycle. The experimental results demonstrate the effectiveness of the proposed CycleVAE-based VC, which yields higher accuracy of converted spectra, generates latent features with higher correlation degree, and significantly improves the quality and conversion accuracy of the converted speech.

show abstract

Section: Introductionmentioning

confidence: 99%

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Tobing

Hayashi

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…The proposed training algorithm does not use pre-stored speakers' voice data, unlike one-to-many non-parallel VC [9] and Kinnunen et al's work [12]. One of our future directions is to use the pre-stored speakers' voice data to improve speaker individuality of the converted voice (e.g., using transfer learning [17]).…”

Section: Discussionmentioning

confidence: 99%

“…In one-to-many VC, i.e., VC from a source speaker to any arbitrary speakers, G(·) is often trained using multi-speaker corpora in advance, and is adapted using the specific targeted speaker's voice data. This paper utilizes the d-vector-based model adaptation method [9]. The method employs another DNN that estimates a d-vector (i.e., DNN-based continuous speaker representation [2]) of the targeted speaker, and feeds it to the VC model to convert the source speaker's features into the targeted speaker's ones.…”

Section: One-to-many Non-parallel Vcmentioning

confidence: 99%

“…The F0 and band-aperiodicity conversions are the same as those in the V2S attack. In one-to-many non-parallel VC, the DNN architectures, training data, and other hyperparameters are the same as those in Saito et al's work [9] that used variational autoencoders [27] conditioned by phonetic posteriorgrams [7] and d-vectors [2] as the VC model.…”

Section: Experimental Evaluation 41 Experimental Setupmentioning

confidence: 99%

See 1 more Smart Citation

V2S attack: building DNN-based voice conversion from automatic speaker verification

Nakamura¹,

Saito²,

Takamichi³

et al. 2019

10th ISCA Workshop on Speech Synthesis (SSW 10)

Self Cite

View full text Add to dashboard Cite

This paper presents a new voice impersonation attack using voice conversion (VC). Enrolling personal voices for automatic speaker verification (ASV) offers natural and flexible biometric authentication systems. Basically, the ASV systems do not include the users' voice data. However, if the ASV system is unexpectedly exposed and hacked by a malicious attacker, there is a risk that the attacker will use VC techniques to reproduce the enrolled user's voices. We name this the "verificationto-synthesis (V2S) attack" and propose VC training with the ASV and pre-trained automatic speech recognition (ASR) models and without the targeted speaker's voice data. The VC model reproduces the targeted speaker's individuality by deceiving the ASV model and restores phonetic property of an input voice by matching phonetic posteriorgrams predicted by the ASR model. The experimental evaluation compares converted voices between the proposed method that does not use the targeted speaker's voice data and the standard VC that uses the data. The experimental results demonstrate that the proposed method performs comparably to the existing VC methods that trained using a very small amount of parallel voice data.

show abstract

“…Recently, neural network approaches are widely studied. Deep generative models like variational auto-encoder (VAE) [18,19] and generative adversarial networks (GANs) [20,21] without requiring parallel data are found effective for VC, though in the same language.…”

Section: Introductionmentioning

confidence: 99%

A Modularized Neural Network with Language-Specific Output Layers for Cross-Lingual Voice Conversion

Zhou

Tian

Yılmaz

et al. 2019

2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

View full text Add to dashboard Cite

This paper presents a cross-lingual voice conversion framework that adopts a modularized neural network. The modularized neural network has a common input structure that is shared for both languages, and two separate output modules, one for each language. The idea is motivated by the fact that phonetic systems of languages are similar because humans share a common vocal production system, but acoustic renderings, such as prosody and phonotactic, vary a lot from language to language. The modularized neural network is trained to map Phonetic PosteriorGram (PPG) to acoustic features for multiple speakers. It is conditioned on a speaker i-vector to generate the desired target voice. We validated the idea between English and Mandarin languages in objective and subjective tests. In addition, mixed-lingual PPG derived from a unified English-Mandarin acoustic model is proposed to capture the linguistic information from both languages. It is found that our proposed modularized neural network significantly outperforms the baseline approaches in terms of speech quality and speaker individuality, and mixed-lingual PPG representation further improves the conversion performance.

show abstract

Non-Parallel Voice Conversion Using Variational Autoencoders Conditioned by Phonetic Posteriorgrams and D-Vectors

Cited by 110 publications

References 9 publications

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

Non-Parallel Voice Conversion with Cyclic Variational Autoencoder

V2S attack: building DNN-based voice conversion from automatic speaker verification

A Modularized Neural Network with Language-Specific Output Layers for Cross-Lingual Voice Conversion

Contact Info

Product

Resources

About