Learning Dynamic Stream Weights For Coupled-HMM-based Audio-visual Speech Recognition

Abstract

With the increasing use of multimedia data in communication technologies, the idea of employing visual information in automatic speech recognition (ASR) has recently gathered momentum. In conjunction with the acoustical information, the visual data enhances the recognition performance and improves the robustness of ASR systems in noisy and reverberant environments. In audio-visual systems, dynamic weighting of audio and video streams according to their instantaneous confidence is essential for reliably and systematically achieving high performance. In this paper, we present a complete framework that allows blind estimation of dynamic stream weights for audio-visual speech recognition based on coupled hidden Markov models (CHMMs). As a stream weight estimator, we consider using multilayer perceptrons and logistic functions to map multidimensional reliability measure features to audiovisual stream weights. Training the parameters of the stream weight estimator requires numerous input-output tuples of reliability measure features and their corresponding stream weights. We estimate these stream weights based on oracle knowledge using an expectation maximization algorithm. We define 31-dimensional feature vectors that combine model-based and signal-based reliability measures as inputs to the stream weight estimator. During decoding, the trained stream weight estimator is used to blindly estimate stream weights. The entire framework is evaluated using the Grid audio-visual corpus and compared to state-of-the-art stream weight estimation strategies. The proposed framework significantly enhances the performance of the audio-visual ASR system in all examined test conditions. Index Terms-Audio-visual speech recognition, coupled hidden Markov model, logistic regression, multilayer perceptron, reliability measure, stream weight.

Robust audiovisual speech recognition using noise-adaptive linear discriminant analysis

“…Optimization of stream weights on a frame-by-frame basis has proven its merit for coupled-HMM systems in [34]. It will be interesting to extend this technique to the presented turbodecoding system, adapting the stream weight according to estimated SNR, observation uncertainty, and model-based reliability measures like dispersion and entropy, in order to also consider the timevarying utility of video information in the process.…”

Comparative Study of Visual Feature for Bimodal Hindi Speech Recognition

“…AVASR system consists of two important units: front-end unit and back-end unit (Abdelaziz et al, 2015). The main purpose of the front-end unit is preprocessing and feature extraction.…”

Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

“…The work was inspired by previous applications of DSWs, which were initially proposed in the context of audiovisual ASR. Pioneering work in this regard was conducted in [20] for probabilistic inference incorporating DSWs into hidden Markov models (HMMs). Compared to conventional Bayesian fusion techniques, this allows to rapidly adapt the estimation process.…”