End-to-end acoustic modeling using convolutional neural networks for HMM-based automatic speech recognition

Abstract

In hidden Markov model (HMM) based automatic speech recognition (ASR) system, modeling the statistical relationship between the acoustic speech signal and the HMM states that represent linguistically motivated subword units such as phonemes is a crucial step. This is typically achieved by first extracting acoustic features from the speech signal based on prior knowledge such as, speech perception or/and speech production knowledge, and, then training a classifier such as artificial neural networks (ANN), Gaussian mixture model that estimates the emission probabilities of the HMM states. Recent advances in machine learning techniques, more specifically in the field of image processing and text processing, have shown that such divide and conquer strategy (i.e., separating feature extraction and modeling steps) may not be necessary. Motivated from these studies, we propose an end-to-end acoustic modeling approach using convolution neural networks (CNNs), where the CNN takes as input raw speech signal and estimates the HMM states class conditional probabilities at the output. Alternately, in this approach the relevant features and the classifier are jointly learned from the raw speech signal. Through ASR studies and analyses on multiple languages and multiple tasks, we show that: (a) the proposed approach yields consistently a better system with fewer parameters when compared to the conventional approach of cepstral feature extraction followed by ANN training, (b) unlike conventional method of speech processing, in the proposed approach the relevant feature representations are learned by first processing the input raw speech at sub-segmental level (≈ 2 ms). Specifically, through an analysis we show that the filters in the first convolution layer automatically learn "in-parts" formant-like information present in the sub-segmental speech, and (c) the intermediate feature representations obtained by subsequent filtering of the first convolution layer output are more discriminative compared to standard cepstral features and could be transferred across languages and domains.

Learning Voice Source Related Information for Depression Detection

“…To better understand the spectral information being modelled by the CNNs, we analysed the cumulative frequency response of the first convolutional layer filters, as done in [39,21]: where N f is the number of filters and F k is the frequency response of filter f k . Fig.…”

Improving Children Speech Recognition through Feature Learning from Raw Speech Signal

“…In the conventional ASR systems ( Fig.1-conventional method), the task of recognizing speech is divided into several subtasks, each of which are optimized independently. In [12,9], an end-to-end acoustic modeling approach was proposed, where both the features and the classifier are jointly learned. As shown in Fig.1-proposed method, the CNN based end-to-end acoustic modeling approach is composed of a feature learning stage, that consists of several convolution layers, and a classifier stage, that consists of fully connected (FC) layers (also called a multi-layer perceptron (MLP)) and an output layer.…”

“…speech signal of about 2ms, which is less than one pitch period. Upon analysis of the filters using two different methods, namely, spectral dictionary based interpretation [12] and guided backpropogation based analysis [13], it was found that the CNN learns to model formant frequency information for phone posterior probability estimation. This is interesting, given the fact that the approach does not assume any specific model for the speech signal.…”

“…In [12], a spectral dictionary interpretation was proposed to understand the information modeled by the first convolution layer. This approach has been applied in other studies, such as [23] and [24], to understand the spectral information modeled by the CNNs.…”

N-Gram Language Model based Continuous Voiced Odia Digit Recognition

“…It was observed that, context dependent untied models overtake others by having lower word error rate and better accuracy. An end-to-end acoustic modeling approach using convolutional neural networks (CNNs) for HMM based ASR was proposed in [29]. In their proposed acoustic modeling approach, the appropriate features and the classifier are mutually learned from the raw speech signals.…”