Unsupervised Representation Learning with Attention and Sequence to Sequence Autoencoders to Predict Sleepiness From Speech

Abstract

Motivated by the attention mechanism of the human visual system and recent developments in the field of machine translation, we introduce our attention-based and recurrent sequence to sequence autoencoders for fully unsupervised representation learning from audio files. In particular, we test the efficacy of our novel approach on the task of speech-based sleepiness recognition. We evaluate the learnt representations from both autoencoders, and conduct an early fusion to ascertain possible complementarity between them. In our frameworks, we first extract Mel-spectrograms from raw audio. Second, we train recurrent autoencoders on these spectrograms which are considered as time-dependent frequency vectors. Afterwards, we extract the activations of specific fully connected layers of the autoencoders which represent the learnt features of spectrograms for the corresponding audio instances. Finally, we train support vector regressors on these representations to obtain the predictions. On the development partition of the data, we achieve Spearman's correlation coefficients of .324, .283, and .320 with the targets on the Karolinska Sleepiness Scale by utilising attention and non-attention autoencoders, and the fusion of both autoencoders' representations, respectively. In the same order, we achieve .311, .359, and .367 Spearman's correlation coefficients on the test data, indicating the suitability of our proposed fusion strategy. CCS CONCEPTS• Computing methodologies → Neural networks.

Demystifying the Role of Natural Language Processing (NLP) in Smart City Applications: Background, Motivation, Recent Advances, and Future Research Directions

“…Autoencoders are unsupervised learning models that are used for sequence-to-sequence modeling [76]. The input is studied by the encoder that generates a code label corresponding to it.…”

3D - Learning Representations From Audio Using Autoencoders

“…Using the conditional probabilities of phrase pairings generated by the RNN Encoder-Decoder as an extra feature in the current log-linear model, the performance of the statistical machine translation system is shown to increase empirically. Shahin Amiriparian et al [3]. For unsupervised representation learning, propose a recurrent sequence to sequence autoencoder.…”

“…To learn a representation (encoding), we utilise an auto encoder, and to categorise predictions, we employ LSTM networks. [2][3], [6-8][16].…”

How to Design a Relevant Corpus for Sleepiness Detection Through Voice?

“…They achieved respective correlations of ρ = 0.369 and ρ = 0.343 while the winner of the challenge, who used Fischer vectors and bag-of-features ( 24 ), achieved a correlation of ρ = 0.387. Since then, two other systems using deep learning have been proposed ( 25 , 26 ), both achieving performances below the winner of the challenge (resp. ρ = 0.317 and ρ = 0.367).…”