Lip-reading via a DNN-HMM hybrid system using combination of the image-based and model-based features

Rahmani, Mohammad Hasan; Almasganj, Farshad

doi:10.1109/pria.2017.7983045

Cited by 28 publications

(10 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…viseme appearance features, were utilized in our system, resulting in the base algorithm of accuracy 69.8%. It should be noted that our deep CNN classifier outperforms the CNN model presented in [7], where accuracy of 76.6% is attained in [7], in spite of using large viseme dataset. In our work, the accuracy is www.ijacsa.thesai.org increased by about 21%.…”

Section: Discussionmentioning

confidence: 83%

See 1 more Smart Citation

Novel Algorithm Utilizing Deep Learning for Enhanced Arabic Lip Reading Recognition

Khafaga¹,

Mahmoud²,

Alghamdi³

et al. 2021

IJACSA

View full text Add to dashboard Cite

Computerized lip reading is the science of translating visemes and oral lip reading into written text, where visemes are lip movement without sound. Video processing is applied for the recognition of those visemes. Previous research developed automated systems to for computerized lip reading recognition to be hearing-impaired aid. Many challenges face such an automation process, including insufficient training datasets. Also, speaker-dependency is one of the challenges that are faced. Real-time applications which respond within a specific time period are also widely required. Real-time human computer interaction are systems which require real time response. Response time for human computer interaction is measured by number of elapsed video frames. Video processing of lip reading necessitates real-time implementation. There are applications for viseme recognition, as an aid for deaf people, video games with human computer interaction, and surveillance systems. In this paper, a real-time viseme recognition system is introduced. In order to enhance existing methods to overcome these pitfalls, this paper proposed a computerized lip reading technique based on feature extraction. We utilized blocks arrangement techniques to reach a near-optimal appearance feature extraction technique. A Deep Neural Network is utilized to enhance recognition. The benchmark dataset SAVE, for Arabic visemes, is employed in this research, and high viseme recognition accuracies are achieved. The described computerized lip reading recognition technique is advantageous for the hearing impaired and for other speakers in noisy environments.

show abstract

Section: Discussionmentioning

confidence: 83%

“…The authors in [7], proposed lip viseme modalities through multimodal learning methods. In [8], two deep CNNs are trained using text labels and video frames and their final layers are fused to extract mutual features to be classified by a deep network.…”

Section: Introductionmentioning

confidence: 99%

Novel Algorithm Utilizing Deep Learning for Enhanced Arabic Lip Reading Recognition

Khafaga¹,

Mahmoud²,

Alghamdi³

et al. 2021

IJACSA

View full text Add to dashboard Cite

show abstract

“…Ngiam et al [6] achieved a 68.70% word recognition rate using an RBM-Autoencoder. Rahmani [141] extracted deep bottleneck features, and then used a GMM-HMM for the language model to achieve a WAR of 63.40%. Petridis et al [78] achieved a WAR of 78.60% using the dual flow method.…”

Section: Performances In Lip-readingmentioning

confidence: 99%

Deep Learning-Based Automated Lip-Reading: A Survey

Fenghour¹,

Chen

Guo³

et al. 2021

IEEE Access

View full text Add to dashboard Cite

A survey on automated lip-reading approaches is presented in this paper with the main focus being on deep learning related methodologies which have proven to be more fruitful for both feature extraction and classification. This survey also provides comparisons of all the different components that make up automated lip-reading systems including the audio-visual databases, feature extraction, classification networks and classification schemas. The main contributions and unique insights of this survey are: 1) A comparison of Convolutional Neural Networks with other neural network architectures for feature extraction; 2) A critical review on the advantages of Attention-Transformers and Temporal Convolutional Networks to Recurrent Neural Networks for classification; 3) A comparison of different classification schemas used for lip-reading including ASCII characters, phonemes and visemes, and 4) A review of the most up-to-date lip-reading systems up until early 2021.

show abstract

“…A HMM model which predicts septic shock for ICU patients [10]. Rehabilitation of a deaf person is done by a DNN-HMM hybrid system for lip-reading and audio visual speech recognition (AVSR) [11], fall detection and real-life home monitoring for senior citizens [12], emotion classification by a combined SVM-HMM classifier to recognize human emotion states based on EEG signals [13]. Already a stacked HMM model has found its applications in robotics for motion intention recognition based on motion trajectories [14].…”

Section: Introductionmentioning

confidence: 99%

Detection of Different Brain Diseases from EEG Signals Using Hidden Markov Model

Rabbani¹,

Islam²

2019

IJIGSP

View full text Add to dashboard Cite

The brain imaging device, Electroencephalography (EEG) provides several advantages over other brain signals like Functional Nearinfrared Spectroscopy (fNIRS) and Functional Magnetic Resonance Imaging (fMRI). It is non-invasive and easily applicable. EEG provides high temporal resolution with a low setup cost. EEG signals of several subjects which record electric potential caused by neurons firing in the brain are undergone a Hidden Markov Model (HMM) classification technique. We are particularly interested to detect the brain diseases from EEG signals by an HMM probabilistic model. This HMM model is built with a given initial probability matrix of five different states, namely, epilepsy, seizure, dementia, stroke and normality. The transition probability matrix is updated after each iteration of parameter estimation using Baum-Welch algorithm (B-W algorithm).

show abstract

Lip-reading via a DNN-HMM hybrid system using combination of the image-based and model-based features

Cited by 28 publications

References 8 publications

Novel Algorithm Utilizing Deep Learning for Enhanced Arabic Lip Reading Recognition

Novel Algorithm Utilizing Deep Learning for Enhanced Arabic Lip Reading Recognition

Deep Learning-Based Automated Lip-Reading: A Survey

Detection of Different Brain Diseases from EEG Signals Using Hidden Markov Model

Contact Info

Product

Resources

About