Designing relevant features for visual speech recognition

Benhaim, Eric; Sahbi, Hichem; Vitte, Guillaume

doi:10.1109/icassp.2013.6638089

Cited by 13 publications

(9 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that, in practice, we use string kernel maps for Φ [7]. Details about the design of these kernel maps, out of the main scope of this paper, are deliberately omitted and can be found in [7].…”

Section: Kernel-based Unary Potentialmentioning

confidence: 99%

Continuous visual speech recognition for audio speech enhancement

Benhaim

Sahbi

Vittey

2015

2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

We introduce in this paper a novel non-blind speech enhancement procedure based on visual speech recognition (VSR). The latter is based on a generative process that analyzes sequences of talking faces and classifies them into visual speech units known as visemes. We use an effective graphical model able to segment and label a given sequence of talking faces into a sequence of visemes. Our model captures unary potential as well as pairwise interaction; the former models visual appearance of speech units while the latter models their interactions using boundary and visual language model activations. Experiments conducted on a standard challenging dataset, show that when feeding the results of VSR to the speech enhancement procedure, it clearly outperforms baseline blind methods as well as related work.

show abstract

Section: Kernel-based Unary Potentialmentioning

confidence: 99%

Continuous visual speech recognition for audio speech enhancement

Benhaim

Sahbi

Vittey

2015

2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Evano and Besacier [8] investigated liveness verification based upon an analysis of the synchronicity of visual and audio features and reported an Equal Error Rate of 14.5% using the XM2VTS dataset. In [10] a liveness verification system based on only using visual information was proposed that is based on speech recognition with an SVM (support vector machine) to recognize digits that had been individually segmented. A speech recognition rate of 68% was reported on the XM2VTS dataset, using the approach in [10] with only the visual modality.…”

Section: Introductionmentioning

confidence: 99%

“…In [10] a liveness verification system based on only using visual information was proposed that is based on speech recognition with an SVM (support vector machine) to recognize digits that had been individually segmented. A speech recognition rate of 68% was reported on the XM2VTS dataset, using the approach in [10] with only the visual modality. In this paper, the aim is to show an improvement over previous works through the use of deep learning.…”

Section: Introductionmentioning

confidence: 99%

Liveness Verification Using Deep Neural Network Based Visual Speech Recognition

McShane¹,

Stewart²

2018

IJMIP

View full text Add to dashboard Cite

We present a novel approach to liveness verification based on visual speech recognition within a challenge-based framework which has the potential to be used on mobile devices to prevent replay or spoof attacks during Facebased liveness verification. The system uses visual speech recognition and determines liveness based on the Levenshtein Distance between a randomly generated challenge phrase and the hypothesis utterances from the visual speech recognizer. A Deep learning-based approach to visual speech recognition is used to improve upon the state of the art for the use of visual speech recognition for liveness verification. A comparison of a number of different deep neural networks is presented. The performance of these deep neural networks on a visual speech recognition task (digit recognition) is used to determine which is the most suitable for use in a liveness verification system. Experiments are performed using Long short-term memory (LSTM), bi-directional long short-term memory (BLSTM) and time delay neural networks (TDNN) with a Gaussian mixture model (GMM) being used to provide a baseline for comparison. The best performing of these is then used for determining the performance of a visual speech-based liveness verification system on a variety of phrase lengths.

show abstract

“…The studies carried out by Benhaim et al [67] using the CUAVE database reported a speech recognition accuracy of 85% in speaker independent experiments. For visual features the approach used histogram-based descriptors around twelve lip landmarks determined using an AAM fitting technique and the classification involved multiple kernel learning and SVM.…”

Section: Comparison With Other Studiesmentioning

confidence: 99%

“…Lip reading is the ability or skill to understand speech through information gleaned from the lower part of face, typically by following lip, tongue and jaw movement patterns. Speechreading may include lip reading information, but may provide additional understanding of the speech by interpreting whole face expressions, gestures and body language [4][5][6], as well as environmental conditions, such as specific characteristics of the speaker and the time and physical location at which the conversation took place [7].…”

Section: Introductionmentioning

confidence: 99%