Vector-Quantized Neural Networks for Acoustic Unit Discovery in the ZeroSpeech 2020 Challenge

Niekerk, Benjamin van; Nortje, Leanne; Kamper, Herman

doi:10.21437/interspeech.2020-1693

Cited by 89 publications

(75 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, the state-of-the-art VQ-VAE method [18] is first described, and then we introduce the proposed LLTs model.…”

Section: Methodsmentioning

confidence: 99%

“…For the baseline VQ-VAE method, we chose the same network architecture in [18], which is a quite competitive model in VQ-VAE based voice conversion. The network architecture of the proposed LLTs method is illustrated in Fig 3, which is similar to the baseline model except for the multi-head attention module.…”

Section: Network Architecture and Implementation Detailsmentioning

confidence: 99%

“…However, VAE tends to generate over-smoothing speeches due to the over-simplified assumptions over the latent variables. Recently, Vector quantized VAE (VQ-VAE) [17,18,19] is proposed to learn the discrete latent representation instead of the continuous one in original VAE through vector quantization. Although VQ-VAE has a strong capability of removing speaker information, it also degrades the quality of the converted speeches.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Non-Parallel Many-To-Many Voice Conversion Using Local Linguistic Tokens

Wang

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The VQ-VAE based voice conversion models have lately received increasing attention in non-parallel many to many voice conversion, where the encoder extracts the speakerinvariant linguistic content from the input speech using vector quantization and the decoder produces the target speech from the encoder output, conditioned on the target speaker representation. However, it is challenging for the encoder to find a proper balance between removing the speaker information and preserving the linguistic content, which degrades the converted speech quality. To address this issue, we propose the Local Linguistic Tokens (LLTs) model to learn high-quality speaker-invariant linguistic embeddings using the multi-head attention module, which has shown great success in extracting speaking style embeddings in Global Style Tokens (GSTs). Instead of vector quantization, the multihead attention module makes the encoder preserve more linguistic content to enhance the converted speech quality. Both objective and subjective experimental results revealed that, compared with the state-of-the-art VQ-VAE model, the proposed LLTs model achieved significantly better speech quality and comparable speaker similarity. The converted samples are available online for listening. 1

show abstract

“…In this section, the state-of-the-art VQ-VAE method [18] is first described, and then we introduce the proposed LLTs model.…”

Section: Methodsmentioning

confidence: 99%

Section: Network Architecture and Implementation Detailsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Non-Parallel Many-To-Many Voice Conversion Using Local Linguistic Tokens

Wang

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…To this end an optimal cluster assignment problem was solved for each training sample. This approach was refined in [18] for phoneme segmentation in a VQ-VAE and VQ-CPC [19]. Finally, [20] have proposed to use slowness penalty and run-length encoding of the latent representation of a VQ-VAE.…”

Section: Related Workmentioning

confidence: 99%

Aligned Contrastive Predictive Coding

Marxer¹,

Chorowski²,

Ciesielski³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss, to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than the sequence of upcoming representations to which they will be aligned. In this way, the prediction network solves a simpler task of predicting the next symbols, but not their exact timing, while the encoding network is trained to produce piece-wise constant latent codes. We evaluate the model on a speech coding task and demonstrate that the proposed Aligned Contrastive Predictive Coding (ACPC) leads to higher linear phone prediction accuracy and lower ABX error rates, while being slightly faster to train due to the reduced number of prediction heads.

show abstract

“…To ease the burden of collecting parallel utterances, nonparallel VC has been developed. There are two major nonparallel VC techniques, namely phonetic posteriorgram (PPG)-based VC methods [6,5] and autoencoderbased VC methods including those using the variational autoencoder (VAE) [7,8,9,10], vector-quantized VAE (VQ-VAE) [11,12,13,14,15], and generative adversarial network (GAN) [16,17,18,19]. For the PPG-based VC method, the PPG vector is first estimated using a preliminarily trained automatic speech recognition (ASR) system.…”

Section: Introductionmentioning

confidence: 99%

Crank: An Open-Source Software for Nonparallel Voice Conversion Based on Vector-Quantized Variational Autoencoder

Kobayashi

Huang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this paper, we present an open-source software for developing a nonparallel voice conversion (VC) system named crank. Although we have released an open-source VC software based on the Gaussian mixture model named sprocket in the last VC Challenge, it is not straightforward to apply any speech corpus because it is necessary to prepare parallel utterances of source and target speakers to model a statistical conversion function. To address this issue, in this study, we developed a new open-source VC software that enables users to model the conversion function by using only a nonparallel speech corpus. For implementing the VC software, we used a vector-quantized variational autoencoder (VQVAE). To rapidly examine the effectiveness of recent technologies developed in this research field, crank also supports several representative works for autoencoder-based VC methods such as the use of hierarchical architectures, cyclic architectures, generative adversarial networks, speaker adversarial training, and neural vocoders. Moreover, it is possible to automatically estimate objective measures such as mel-cepstrum distortion and pseudo mean opinion score based on MOSNet. In this paper, we describe representative functions developed in crank and make brief comparisons by objective evaluations.

show abstract

Vector-Quantized Neural Networks for Acoustic Unit Discovery in the ZeroSpeech 2020 Challenge

Cited by 89 publications

References 29 publications

Non-Parallel Many-To-Many Voice Conversion Using Local Linguistic Tokens

Non-Parallel Many-To-Many Voice Conversion Using Local Linguistic Tokens

Aligned Contrastive Predictive Coding

Crank: An Open-Source Software for Nonparallel Voice Conversion Based on Vector-Quantized Variational Autoencoder

Contact Info

Product

Resources

About