Visual speech recognition using active shape models and hidden Markov models

Luettin, Juergen; Thacker, Neil A.; Beet, S.W.

doi:10.1109/icassp.1996.543246

Cited by 78 publications

(46 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…An early approach that exploits the characteristic temporal signature of faces based on partially recurrent neural networks trained over sequences of facial images was first introduced by Gong, Psarrou, Katsoulis, and Palavouzis (1994). Initial experiments conducted by Luettin and colleagues suggested that spatiotemporal models (HMMs) trained on sequences of lip motion during speech could be useful for speaker recognition (Luettin, Thacker, & Beet, 1996). However, beyond this early experiment, the use of spatiotemporal cues for the identification of people in computer vision has remained relatively unexplored (Gong, McKenna, & Psarrou, 2000).…”

Section: Computer Vision Models For the Processing Of Dynamic Facesmentioning

confidence: 99%

Elements for a Neural Theory of the Processing of Dynamic Faces

Serre¹,

Giese²

2010

Dynamic Faces

View full text Add to dashboard Cite

Face recognition has been a central topic in computer vision for at least two decades and progress in recent years has been significant. Automated face recognition systems are now widespread in applications ranging from surveillance to personal computers. In contrast, only a handful of neurobiologically plausible computational models have been proposed to try to explain the processing of faces in the primate cortex (e.g., Giese & Leopold, 2005;Jiang et al., 2006), and no such model has been applied specifically in the context of dynamic faces.There is a need for an integrated computational theory of dynamic face processing that could integrate and summarize evidence obtained with di¤erent experimental methods, from single-cell physiology to fMRI, MEG, and ultimately behavior and psychophysics. At the same time, physiologically plausible models capable of processing real video sequences constitute a plausibility proof for the computational feasibility of di¤erent hypothetical neural mechanisms.In this review chapter we will first discuss computer vision models for the processing of dynamic faces; these do not necessarily try to reproduce biological data but may suggest relevant computational principles. We then provide an overview of computational neuroscience models for the processing of static faces and dynamic body stimuli. We further highlight specific elements from our own work that are likely to be relevant for the processing of dynamic face stimuli. The last section discusses open problems and critical experiments from the viewpoint of neural computational approaches to the processing of dynamic faces. Computational Models for the Processing of Faces and BodiesThe following section reviews work in computer vision as well as neural and psychological models for the recognition of static faces. In addition, we discuss neurobiological models for the processing of dynamic bodies. It seems likely that some of the computational principles proposed in the context of these models might also be relevant for the processing of dynamic faces.

show abstract

Section: Computer Vision Models For the Processing Of Dynamic Facesmentioning

confidence: 99%

Elements for a Neural Theory of the Processing of Dynamic Faces

Serre¹,

Giese²

2010

Dynamic Faces

View full text Add to dashboard Cite

show abstract

“…The first is a top-down approach, where an a priori lipshape representation framework is embedded in a model; for example, active shape models (ASMs) [31] and active appearance models (AAMs) [11]. ASMs and AAMs extract higher-level, model-based features derived from the shape and appearance of mouth area images.…”

Section: Visual Feature Extraction Mechanismsmentioning

confidence: 99%

Audio-visual speech recognition using deep learning

et al. 2014

View full text Add to dashboard Cite

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signalto-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.

show abstract

“…However, these approaches cannot adapt to considerable geometric changes in lips in every frame. In the model-based approach, active shape model (6) and genetic snakes (8) (25) , that is, an improved version of Snakes (26) , have been proposed. These approaches have some constraints, such as, the target image must be a face region of fixed pose and a subject wears a camera to get a mouth image, etc.…”

Section: Related Workmentioning

confidence: 99%

High Speed Genetic Lips Detection by Dynamic Search Domain Control

et al. 2007

View full text Add to dashboard Cite

In this paper, high-speed size and orientation invariant lips detection of a talking person in an active scene using template matching and genetic algorithms is proposed (refer to Fig. 1). As part of the objectives, we also try to acquire numerical parameters to represent the lips. The information is very important for many applications, where high performance is required, such as audio-visual speech recognition, and personal mobile devices interfaces. The difficulty in lips detection is mainly due to deformations and geometric changes of the lips during speech and the active scene by free camera motion. In order to enhance the performance in speed and accuracy, initially, the performance is improved on a single still image, that is, the base of video processing.Our proposed system is based on template matching using genetic algorithms (GA). Only one template is prepared per experiment. The template is the closed mouth of a subject, because the application is for personal devices. In our previous study, the main problem was trade-off between search accuracy and search speed. To overcome this problem, we use two methods: scaling window and dynamic search domain control (SD-Control).The transition of the objective value on the proposed system and the no SD-Control are plotted in Fig. 2. At points from (a) to (d) in Figs. 2, the transition of GA exploration is exactly the same in both cases. The reason is that the same random numbers are used in both cases. Between points (d) and (e) in Fig. 2, the differences in the methods appear. This is because, the search domain is controlled from point (d) in Fig. 2. The elite individual of the no SD-Control do not change from point (c) to (g), hence it is concluded that the GA exploration gets trapped into a local optimum. The reason why the exploration was trapped into a local optimum is that the small population loses the population diversity and causes premature convergence.On the other hand, in case of the proposed system, the evolution make good progress from generation = 117 (point from (e) to (i)). This indicates that the GA of the proposed system can escape from the local maximum. The reason for this is that a search domain is reduced by the dynamic SD-Control and the GA can explore the reduced search domain with meticulous detail. This means that the GA in the proposed system performs not only global optimization but also local optimization. The result of our demonstration clearly shows that a trade-off between exploration accuracy and speed is overcome by the dynamic SD-Control.We achieved a lips detection accuracy of 91.33% at an average processing time of 33.70 milliseconds per frame. Stephen Karungaru * * Non-member Minoru Fukumi * * MemberIn this paper, high-speed size and orientation invariant lips detection of a talking person in an active scene using template matching and genetic algorithms is proposed. As part of the objectives, we also try to acquire numerical parameters to represent the lips. The information is very important for many applications, where ...

show abstract

Visual speech recognition using active shape models and hidden Markov models

Cited by 78 publications

References 12 publications

Elements for a Neural Theory of the Processing of Dynamic Faces

Elements for a Neural Theory of the Processing of Dynamic Faces

Audio-visual speech recognition using deep learning

High Speed Genetic Lips Detection by Dynamic Search Domain Control

Contact Info

Product

Resources

About