Recent advances in the automatic recognition of audiovisual speech

Abstract

Abstract-Visual speech information from the speaker's mouth region has been successfully shown to improve noise robustness of automatic speech recognizers, thus promising to extend their usability into the human computer interface. In this paper, we review the main components of audio-visual automatic speech recognition and present novel contributions in two main areas: First, the visual front end design, based on a cascade of linear image transforms of an appropriate video region-of-interest, and subsequently, audio-visual speech integration. On the later topic, we discuss new work on feature and decision fusion combination, the modeling of audio-visual speech asynchrony, and incorporating modality reliability estimates to the bimodal recognition process. We also briefly touch upon the issue of audiovisual speaker adaptation. We apply our algorithms to three multi-subject bimodal databases, ranging from small-to largevocabulary recognition tasks, recorded at both visually controlled and challenging environments. Our experiments demonstrate that the visual modality improves automatic speech recognition over all conditions and data considered, however less so for visually challenging environments and large vocabulary tasks.

Visual Speech Recognition Using PCA Networks and LSTMs in a Tandem GMM-HMM System

“…Typically, this is done by applying a transformation such as the discrete cosine transform (DCT) and/or a dimensionality reduction technique such as the linear discriminant analysis (LDA) to the ROI, possibly in combination with a principle component analysis (PCA) or a maximum-likelihood linear transform (MLLT) [23]. A common feature post-processing technique involves a chain of LDAs and MLLTs on concatenated frames, the so-called HiLDA [22].…”

Geometrical-based lip-reading using template probabilistic multi-dimension dynamic time warping

“…Lip visual features are generally grouped into three categories [31][32][33]: (a) appearance-based features; (b) shape-based features and (c) a combination of both appearance and shape features.…”

“…Since only a small part of the vocal tract is visible when we speak, only partial physical information is available regarding the generation of visemes and not all can be mapped to a unique phoneme [32], the basic unit of speech in the audio domain.…”

Read My Lips: Continuous Signer Independent Weakly Supervised Viseme Recognition

“…Lan et al [8] achieve an accuracy of 45% on their challenging 12 speakers audio-visual corpus. A good overview of the field is given in [14] and [15]. Neti et al [16] present audio-visual but also visual only recognition results.…”