Convolutional RNN: An enhanced model for extracting features from sequential data

Abstract

Traditional convolutional layers extract features from patches of data by applying a non-linearity on an affine function of the input. We propose a model that enhances this feature extraction process for the case of sequential data, by feeding patches of the data into a recurrent neural network and using the outputs or hidden states of the recurrent units to compute the extracted features. By doing so, we exploit the fact that a window containing a few frames of the sequential data is a sequence itself and this additional structure might encapsulate valuable information. In addition, we allow for more steps of computation in the feature extraction process, which is potentially beneficial as an affine function followed by a non-linearity can result in too simple features. Using our convolutional recurrent layers, we obtain an improvement in performance in two audio classification tasks, compared to traditional convolutional layers. Tensorflow code for the convolutional recurrent layers is publicly available in https://github.com/cruvadom/Convolutional-RNN.

Attentive Convolutional Neural Network Based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech

“…DL has been shown to significantly boost emotion recognition performance [17,18,19,20,21,22]. Recently, several papers [23,24] presented CNNs in combination with Long Short-Term Memory models (LSTM) to improve speech emotion recognition based on log Mel filter-banks (logMel) or raw signal. [24] demonstrated an end-to-end training from raw signal.…”

Detecting Alzheimer’s Disease Using Gated Convolutional Neural Network from Audio Data

“…In this study, we propose a non-linguistic approach for detecting AD using acoustic features from speech data. Inspired by numerous successes with deep learning for paralinguistic tasks such as for emotion recognition [10,11], we employ convolutional neural networks with a gating mechanism.…”

Comparison of Deep Neural Networks and Deep Hierarchical Models for Spatio-Temporal Data

“…Although DNNs can be used with spatio-temporal data , they are not always appropriate because they do not naturally accommodate dependence structures that occur in time and space. However, given the modularity of CNNs and RNNs (i.e., they are easily "stacked" to make deeper models) it is no surprise that they can easily be combined in different ways to produce deep hybrid models for spatio-temporal data, such as video image processing and image captioning (e.g., Keren and Schuller, 2016;Tong and Tanaka, 2018). For example, images in a video can be reduced by a CNN to find spatial features and the time evolution of these features can then be modeled with an RNN (usually an LSTM).…”