Real-time lip reading system for isolated Korean word recognition

Shin, Jongju; Liu, Jin; Kim, Daijin

doi:10.1016/j.patcog.2010.09.011

Cited by 48 publications

(18 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yes [14][15][16] Yes, [9] Unit choice Yes, [17][18][19][20][21] Yes, [3,4,[22][23][24] Classifier technology Yes, [17,[25][26][27][28] Multiple persons Yes, [29][30][31][32]…”

Section: Video Qualitymentioning

confidence: 99%

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Bear

Harvey

2017

Speech Communication

View full text Add to dashboard Cite

Visemes are the visual equivalent of phonemes. Although not precisely defined, a working definition of a viseme is “a set of phonemes which have identical appearance on the lips”. Therefore a phoneme falls into one viseme class but a viseme may represent many phonemes: a many to one mapping. This mapping introduces ambiguity between phonemes when using viseme classifiers. Not only is this ambiguity damaging to the performance of audio-visual classifiers operating on real expressive speech, there is also considerable choice between possible mappings. In this paper we explore the issue of this choice of viseme-to-phoneme map. We show that there is definite difference in performance between viseme-to-phoneme mappings and explore why some maps appear to work better than others. We also devise a new algorithm for constructing phoneme-to-viseme mappings from labeled speech data. These new visemes, ‘Bear’ visemes, are shown to perform better than previously known units

show abstract

“…Yes [14][15][16] Yes, [9] Unit choice Yes, [17][18][19][20][21] Yes, [3,4,[22][23][24] Classifier technology Yes, [17,[25][26][27][28] Multiple persons Yes, [29][30][31][32]…”

Section: Video Qualitymentioning

confidence: 99%

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Bear

Harvey

2017

Speech Communication

View full text Add to dashboard Cite

show abstract

“…Jongju Shin et al [2] proposes a real-time lip reading system (consisting of a lip detector, lip tracker, lip activation detector, and word classifier), which can recognize isolated Korean words. Lip detection is performed in several stages: face detection, eye detection, mouth detection, mouth end-point detection, and active appearance model (AAM) fitting.…”

Section: Visual Speech Recogntionmentioning

confidence: 99%

“…In the low noise-level environments, the word correct rate (WCR) for audio-only ASR system is over 95%. In noisy environments however, the WCR is significantly reduced [2,3].…”

Section: Introductionmentioning

confidence: 99%

Automatic bimodal audiovisual speech recognition: A review

Kandagal

Udayashankara²

2014

2014 International Conference on Contemporary Computing and Informatics (IC3I)

View full text Add to dashboard Cite

Human computer interaction (HCI) is very crucial in our day-to-day activity. Speech is one of the essential and intuitive ways to interact with machines such as Smartphone, which has multiple sensors as microphone, camera, etc. An efficient performance speech recognition system improves interaction between man and machines by making latter more receptive to user needs. Such system has Automatic speech recognition (ASR) engine, which is facing a unique challenge of accuracy in recognition rate. By integrating acoustic signal feature vectors with the visual features, a more robust audiovisual speech recognition engine (AVSR) could be developed for real environmental scenarios. This paper presents past research and development in the field of ASR and AVSR technologies. It describes key technological perspective and admiration of the fundamental progress in ASR and AVSR. The objective of this review is to summarize and compare some of the well-known methods experimented by previous researchers, and to conclude with direction on future research proficiency in HCI system using ASR and AVSR engine.

show abstract

“…In our opinion, machine learning is an attractive alternative for feature pooling. Today the field of machine learning and pattern recognition finds applications not only in the traditional fields like speech recognition [29,67] but also in new and emerging research areas (for example, isolated word recognition [23] using lip reading). Machine learning has also been used for many image processing applications such as image classification [56]; image segmentation [3,52], which is often used in many video and computer vision applications such as object localization/tracking/recognition, signal compression, and image retrieval [47]; image watermarking [54]; handwriting recognition [9]; age estimation from facial images [17]; object detection [59]; sketch recognition [69]; texture classification [75], etc.…”

Section: Signal Processing Based Approach For Iqamentioning

confidence: 99%

Scalable image quality assessment with 2D mel-cepstrum and machine learning approach

Narwaria

Wang

Çetin

2012

Pattern Recognition

View full text Add to dashboard Cite

a b s t r a c tMeasurement of image quality is of fundamental importance to numerous image and video processing applications. Objective image quality assessment (IQA) is a two-stage process comprising of the following: (a) extraction of important information and discarding the redundant one, (b) pooling the detected features using appropriate weights. These two stages are not easy to tackle due to the complex nature of the human visual system (HVS). In this paper, we first investigate image features based on two-dimensional (2D) mel-cepstrum for the purpose of IQA. It is shown that these features are effective since they can represent the structural information, which is crucial for IQA. Moreover, they are also beneficial in a reduced-reference scenario where only partial reference image information is used for quality assessment. We address the second issue by exploiting machine learning. In our opinion, the well established methodology of machine learning/pattern recognition has not been adequately used for IQA so far; we believe that it will be an effective tool for feature pooling since the required weights/ parameters can be determined in a more convincing way via training with the ground truth obtained according to subjective scores. This helps to overcome the limitations of the existing pooling methods, which tend to be over simplistic and lack theoretical justification. Therefore, we propose a new metric by formulating IQA as a pattern recognition problem. Extensive experiments conducted using six publicly available image databases (totally 3211 images with diverse distortions) and one video database (with 78 video sequences) demonstrate the effectiveness and efficiency of the proposed metric, in comparison with seven relevant existing metrics.

show abstract

Real-time lip reading system for isolated Korean word recognition

Cited by 48 publications

References 22 publications

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Phoneme-to-viseme mappings: the good, the bad, and the ugly

Automatic bimodal audiovisual speech recognition: A review

Scalable image quality assessment with 2D mel-cepstrum and machine learning approach

Contact Info

Product

Resources

About