Lip-Reading Technique Using Spatio-Temporal Templates and Support Vector Machines

Abstract

Abstract. This paper presents a lip-reading technique to identify the unspoken phones using support vector machines. The proposed system is based on temporal integration of the video data to generate spatiotemporal templates (STT). 64 Zernike moments (ZM) are extracted from each STT. This work proposes a novel feature selection algorithm to reduce the dimensionality of the 64 ZM to 12 features. The proposed technique uses the shape of probability curve as a goodness measure for optimal feature selection. The feature vectors are classified using nonlinear support vector machines.Such a system could be invaluable when it is important to communicate without making a sound, such as giving passwords when in public spaces.

Real-time lip reading system for isolated Korean word recognition

“…Since the ASR system needs to treat various lengths of temporal features, dynamic time warping methods [13] or hidden Markov models (HMM) [14,15] have been widely used to handle the temporal data. Other researchers have used artificial neural networks (ANN) [16,17] and support vector machine (SVM) [18] that are renowned for its excellent generalization performance. However, these methods are not ideal for lip reading applications because they require a huge amount of training data, and would require extensive retraining any time a new word class is added to the database.…”

Automatic visual speech segmentation and recognition using directional motion history images and Zernike moments

“…Various variants of HMMs have also been used for audio-visual ASR, such as HMMs with non-Gaussian continuous observation probabilities [39]. Moreover, additional methods to overcome the difference in the speed of speaking for classification have been employed in audio-visual ASR systems, such as dynamic time warping (DTW), used by Petajan [4] are computationally expensive and inaccurate, while other classifiers that allow the difference among speakers to be considered for classifying the visual data have used artificial neural networks (ANN) [40,41], hybrid ANN-DTW systems [42], hybrid ANN-HMM [43] and recently the support vector machines (SVM) [44]. SVM is based on the structural risk minimization principle in contrast to empirical risk minimization on which many classifiers are based.…”