A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning

Abstract

Probabilistic Latent Variable Models (LVMs) provide an alternative to self-supervised learning approaches for linguistic representation learning from speech. LVMs admit an intuitive probabilistic interpretation where the latent structure shapes the information extracted from the signal. Even though LVMs have recently seen a renewed interest due to the introduction of Variational Autoencoders (VAEs), their use for speech representation learning remains largely unexplored. In this work, we propose Convolutional Deep Markov Model (ConvDMM), a Gaussian state-space model with non-linear emission and transition functions modelled by deep neural networks. This unsupervised model is trained using black box variational inference. A deep convolutional neural network is used as an inference network for structured variational approximation. When trained on a large scale speech dataset (LibriSpeech), ConvDMM produces features that significantly outperform multiple self-supervised feature extracting methods on linear phone classification and recognition on the Wall Street Journal dataset. Furthermore, we found that ConvDMM complements self-supervised methods like Wav2Vec and PASE, improving on the results achieved with any of the methods alone. Lastly, we find that ConvDMM features enable learning better phone recognizers than any other features in an extreme low-resource regime with few labelled training examples.

Textless Speech Emotion Conversion using Discrete and Decomposed Representations

“…Reconstruction is often implemented in the form of auto-encoding (van den Oord et al, 2017), where speech is first encoded into a low-dimensional space, and then decoded back to speech. Various constraints can be imposed on the encoded space, such as temporal smoothness (Ebbers et al, 2017;Glarner et al, 2018;Khurana et al, 2020), discreteness (Ondel et al, 2016;van den Oord et al, 2017), and presence of hierarchy (Hsu et al, 2017), information bottlenecks for speech representation decomposition (Qian et al, 2020(Qian et al, , 2021. Prediction-based approaches task a model with predicting information of unseen speech based on its context.…”

Aligned Contrastive Predictive Coding

“…This makes ACPC somewhat similar to an online variant of pseudo-labeling [21]: the prediction network produces pseudolabels which after a forcealignment serve as targets for the encoder. [22] can be seen as another top-down approach which models Markov-based latent transitions and emissions with neural networks, in contrast ACPC does not model explicitly the representation dynamics.…”

Efficient Generative Modelling of Protein Structure Fragments using a Deep Markov Model

“…Related stochastic sequential neural models were reported by Fraccaro et al (2016) and Chung et al (2015). Published applications of DMMs include natural language processing tasks (Khurana et al, 2020), inference of time series data (Zhi-Xuan et al, 2020), and human pose forecasting (Toyer et al, 2017). Our application of the DMM and the modifications made to the standard model will be described in section 3.3.…”