Improving Unsupervised Subword Modeling via Disentangled Speech Representation Learning and Transformation

Abstract

This study tackles unsupervised subword modeling in the zeroresource scenario, learning frame-level speech representation that is phonetically discriminative and speaker-invariant, using only untranscribed speech for target languages. Frame label acquisition is an essential step in solving this problem. High quality frame labels should be in good consistency with golden transcriptions and robust to speaker variation. We propose to improve frame label acquisition in our previously adopted deep neural network-bottleneck feature (DNN-BNF) architecture by applying the factorized hierarchical variational autoencoder (FHVAE). FHVAEs learn to disentangle linguistic content and speaker identity information encoded in speech. By discarding or unifying speaker information, speaker-invariant features are learned and fed as inputs to DPGMM frame clustering and DNN-BNF training. Experiments conducted on ZeroSpeech 2017 show that our proposed approaches achieve 2.4% and 0.6% absolute ABX error rate reductions in acrossand within-speaker conditions, comparing to the baseline DNN-BNF system without applying FHVAEs. Our proposed approaches significantly outperform vocal tract length normalization in improving frame labeling and subword modeling.

Unsupervised Subword Modeling Using Autoregressive Pretraining and Cross-Lingual Phone-Aware Modeling

“…A typical self-supervised representation learning model is the vector-quantized variational autoencoder (VQ-VAE) [14], which achieved a fairly good performance in ZeroSpeech 2017 [34] and 2019 [9], and has become more widely adopted [35]- [37] in the latest ZeroSpeech 2020 challenge [38]. Other selfsupervised learning algorithms such as factorized hierarchical VAE (FHVAE) [39], contrastive predictive coding (CPC) [22] and APC [26] were also extensively investigated in unsupervised subword modeling [27], [35], [40], [41].…”

“…Heck et al extended this approach by applying unsupervised speaker adaptation, which performed the best in ZeroSpeech 2017 [3]. In a recent study [13], a two-stage bottleneck feature (BNF) learning framework was proposed. The first stage, i.e., the front-end, used the factorized hierarchical variational autoencoder (FHVAE) [14] to learn speaker-invariant features.…”

Combining Adversarial Training and Disentangled Speech Representation for Robust Zero-Resource Subword Modeling

“…During MFCC reconstruction, a male speaker for each of the two languages is randomly selected as the representative speaker for s-vector unification. Our recent research findings [11] showed that male speakers are more suitable than females in generating speaker-invariant features. The IDs of the selected speakers are 'S015' and 'S002' in English and Surprise respectively.…”

“…By concatenating training utterances of the same speaker into a single sequence for FHVAE training, the learned µ2 is expected to be discriminative to speaker identity. This work considers applying s-vector unification [11] to generate reconstructed feature representation that keeps linguistic content unchanged and is more speaker-invariant than the original representation. Specifically, a representative speaker with his/her s-vector (denoted as µ * 2 ) is chosen from the dataset.…”

“…In this paper, we propose to extend our recent research findings [11] on applying disentangled speech representation learned from factorized hierarchical VAE (FHVAE) models [18] to improve speaker-invariant subword modeling. The contributions made in this study are in several aspects.…”