A Comparison of Discrete Latent Variable Models for Speech Representation Learning

Abstract

Neural latent variable models enable the discovery of interesting structure in speech audio data. This paper presents a comparison of two different approaches which are broadly based on predicting future time-steps or auto-encoding the input signal. Our study compares the representations learned by vqvae and vq-wav2vec in terms of sub-word unit discovery and phoneme recognition performance. Results show that future time-step prediction with vq-wav2vec achieves better performance. The best system achieves an error rate of 13.22 on the ZeroSpeech 2019 ABX phoneme discrimination challenge.

Self-Supervised Speech Representation Learning: A Review

“…A similar insight was obtained in [258], which compared vq-vae and vq-wav2vec with respect to their ability to discover phonetic units. The vq-vae model extracts continuous features from the audio signal; a quantizer then maps them into a discrete space, and a decoder is trained to reconstruct the original audio conditioned on the latent discrete representation and the past acoustic observations.…”

Predictive Neural Speech Coding

“…8 to achieve 3kbps. This is quite different from the diversity loss in Gumbel-Softmax based method [26], where a uniform distribution is enforced on the codeword usage.…”

“…Recently, it is also applied to discrete representation learning [12] and serves as the basis of end-to-end neural audio coding [6]- [11]. As quantization is inherently not differentiable, to enable end-to-end learning in neural audio coding, several ways have been proposed in the literature, including the one with commitment loss in VQ-VAE [12], EMA [12], Gumbel-Softmax based method [26] [27] and the soft-to-hard technique [28]. VQ-VAE [12] approximates the derivative by the identity function that directly copies gradients from the decoder input to the encoder output.…”

“…As discussed in Section II.D, Gumbel-Softmax [26] [27] and soft-to-hard [28] methods introduce the probability of selecting a codeword and thus make rate control feasible. However, Gumbel-Softmax uses a linear projection to select the codeword without explicitly correlating it with the quantization error, as shown in Fig.…”

“…3) Distance-Gumbel-Softmax-based VQ: We compare the Distance-Gumbel-Softmax-based VQ mechanism with the previous Gumbel-Softmax-based method in [26]. Table III shows that at 3kbps, our method outperforms the previous Gumbel-Softmax-based method in all metrics, indicating that the explict injection of distance information helps improve recon- We also show the distribution of the learned codebooks to help understand how the Distance-Gumbel-Softmax-based vector quantization learns.…”