A PCA Based Visual DCT Feature Extraction Method for Lip-Reading

Li, Hong; Yao, Xin; WAN, WAN; Chen, Guihai

doi:10.1109/iih-msp.2006.265008

Cited by 49 publications

(29 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Various feature sets have been tested including DCT coefficients [26], active shape models [27] (ASMs), active appearance models (AAMs) and sieve features [22]. The previous section has described papers in which the recognition performance of some of these features has been studied, and the current finding is that AAM features give the best recognition performance overall [24], despite their poor performance in [6].…”

Section: Active Appearance Modelsmentioning

confidence: 99%

Language Identification Using Visual Features

Newman

Cox

2012

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Abstract-Automatic visual language identification (VLID) is the technology of using information derived from the visual appearance and movement of the speech articulators to identify the language being spoken, without the use of any audio information. This technique for language identification (LID) is useful in situations in which conventional audio processing is ineffective (very noisy environments), or impossible (no audio signal is available). Research in this field is also beneficial in the related field of automatic lip-reading. This paper introduces several methods for visual language identification (VLID). They are based upon audio LID techniques, which exploit language phonology and phonotactics to discriminate languages. We show that VLID is possible in a speaker-dependent mode by discriminating different languages spoken by an individual, and we then extend the technique to speaker-independent operation, taking pains to ensure that discrimination is not due to artefacts, either visual (e.g. skin-tone) or audio (e.g. rate of speaking). Although the low accuracy of visual speech recognition currently limits the performance of VLID, we can obtain an error-rate of < 10% in discriminating between Arabic and English on 19 speakers and using about 30s of visual speech.

show abstract

Section: Active Appearance Modelsmentioning

confidence: 99%

Language Identification Using Visual Features

Newman

Cox

2012

IEEE Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

show abstract

“…Principal Component Analysis (PCA) is one of the first choices, and therefore very popular, and was used in many studies e.g. (Bregler et al, 1993); (Bregler & Konig, 1994); (Duchnowski et al, 1994); (Li et al, 1995); (Tomlinson et al, 1996); (Chiou & Hwang, 1997); (Gray et al, 1997); (Li et al, 1997); (Luettin & Thacker, 1997); (Potamianos et al, 1998); (Dupont & Luettin, 2000); (Hong et al, 2006). The feature definition is based on the notion of eigenfaces or eigenlips which represent the eigenvectors of the training sets.…”

Section: Feature Vectors Definitionmentioning

confidence: 99%

“…The feature definition is based on the notion of eigenfaces or eigenlips which represent the eigenvectors of the training sets. An alternative to PCA, very common as well, is Discrete Cosine Transform (DCT) such as in (Duchnowski et al, 1995);(Prez et al, 2005); (Hong et al, 2006); (Lucey & Potamianos, 2006). Linear Discriminant Analysis (LDA), Maximum Likelihood Data Rotation (MLLT), Discrete Wavelet Transform, Discrete Walsh Transform (Potamianos et al, 1998) are other methods that fit in this class and were used for lip reading.…”

Section: Feature Vectors Definitionmentioning

confidence: 99%

Automatic Visual Speech Recognition

Chitu¹,

Rothkrantz²

2012

Speech Enhancement, Modeling and Recognition- Algorithms and Applications

View full text Add to dashboard Cite

“…The transformation of the image is employed in order to obtain some data reduction. The most popular method for this is Principal Component Analysis (PCA) [6,7]. Other methods which were used as an alternative to PCA are based on discrete cosine transform [8] and discrete wavelet transform.…”

Section: Introductionmentioning

confidence: 99%

Comparison between different feature extraction techniques for audio-visual speech recognition

Chitu

Rothkrantz

Wiggers

et al. 2007

J Multimodal User Interfaces

View full text Add to dashboard Cite

Having a robust speech recognition system that can be relied upon in different environments is a strong requirement for modern systems. However audioonly speech recognition still lacks robustness when the signal to noise ratio decreases. This is especially true when the system is deployed in public spaces or is used for crises situations management where the background noise is expected to be extremely large. The video information is not affected by noise which makes it an ideal candidate for data fusion. The acoustic features have been well defined during the course of the years, the most used features being mel-frequency cepstral coefficiens (MFCCs) or linear predictive coefficients(LPCs). On the visual side, however, there is still much place for improvements. It is still not clear which visual features retain the most speech related information. Until now the visual features used were static features which describe the face of the user at one instance in time only. In the paper [1] the authors have shown that most of the techniques used for extraction of static visual features result in equivalent features or at least the most informative features exhibit this. This means that all techniques describe the same aspect of the visual stream. However the improvement of recognition even though looks promis- ing is still modest. We argue that the main problem of existing methods is that the resulting features contain no information about the motion of the speaker's lips. We present in this paper a new method for extracting useful features from the point of view of speech recognition based on optical flow analysis. The video features extracted using this method are preserving the information about speaker mouth motion. We tested the method on an audio-video database for Dutch language. The Audio-Visual Speech Recognizer(AVSR) used is based on HMMs method and was trained for large vocabulary continuous speech. For completion we also present the method introduced in the paper [2] for extracting static visual features. We will compare these two methods with respect to the induced recognition performance. Another way to recover motion information from static features is to use their first and/or the second derivative as visual features. However this can not always guaranty that the resulted features are physically sound quantities. We will also present for comparison the recognition results based on such features. The evaluation of these methods will be done under different noise conditions. We show that the audio-video recognition based on the true motion features outperforms the other settings in low Signal to Noise Ratio(SNR) conditions.

show abstract

A PCA Based Visual DCT Feature Extraction Method for Lip-Reading

Cited by 49 publications

References 13 publications

Language Identification Using Visual Features

Language Identification Using Visual Features

Automatic Visual Speech Recognition

Comparison between different feature extraction techniques for audio-visual speech recognition

Contact Info

Product

Resources

About