Visual Speech Recognition Using Optical Flow and Hidden Markov Model

Sharma, Usha; Maheshkar, Sushila; Mishra, Achyuta Nand; Kaushik, Rahul

doi:10.1007/s11277-018-5930-z

Cited by 16 publications

(8 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model achieved 71% in word recognition accuracy for the CUAVE dataset [23]. Sharma et al [24] have investigated audio-visual speech recognition for the Hindi language by utilizing the Mel Frequency Cepstral Coefficient (MFCC) for audio features. For visual features, they have used the Optical Flow (OF) and conventional lip localization approach, and then it is followed by a HMM for recognition.…”

Section: Handcrafted Features Based Modelsmentioning

confidence: 99%

Deep Learning-Based Approach for Arabic Visual Speech Recognition

Ullah¹,

Zahid²,

Algarni³

et al. 2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Lip-reading technologies are rapidly progressing following the breakthrough of deep learning. It plays a vital role in its many applications, such as: human-machine communication practices or security applications. In this paper, we propose to develop an effective lip-reading recognition model for Arabic visual speech recognition by implementing deep learning algorithms. The Arabic visual datasets that have been collected contains 2400 records of Arabic digits and 960 records of Arabic phrases from 24 native speakers. The primary purpose is to provide a high-performance model in terms of enhancing the preprocessing phase. Firstly, we extract keyframes from our dataset. Secondly, we produce a Concatenated Frame Images (CFIs) that represent the utterance sequence in one single image. Finally, the VGG-19 is employed for visual features extraction in our proposed model. We have examined different keyframes: 10, 15, and 20 for comparing two types of approaches in the proposed model: (1) the VGG-19 base model and (2) VGG-19 base model with batch normalization. The results show that the second approach achieves greater accuracy: 94% for digit recognition, 97% for phrase recognition, and 93% for digits and phrases recognition in the test dataset. Therefore, our proposed model is superior to models based on CFIs input.

show abstract

Section: Handcrafted Features Based Modelsmentioning

confidence: 99%

Deep Learning-Based Approach for Arabic Visual Speech Recognition

Ullah¹,

Zahid²,

Algarni³

et al. 2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

show abstract

“…Visual feature extraction can be mainly classified as modal-based and imagebased feature extraction. In model-based feature extraction, the Active Appearance Model (AAM) gained more attention [76,80] [75,78,79]. In, audio-visual fusion preferred models are the Hidden Markov Model (HMM) and the Multi-Stream Hidden Markov Model (MSHMM) [75,76,77,78,79].…”

Section: Audio-visual Speech Recognitionmentioning

confidence: 99%

“…In model-based feature extraction, the Active Appearance Model (AAM) gained more attention [76,80] [75,78,79]. In, audio-visual fusion preferred models are the Hidden Markov Model (HMM) and the Multi-Stream Hidden Markov Model (MSHMM) [75,76,77,78,79]. The introduction of deep learning models enhanced the performance of ASR, VSR, and AVSR systems.…”

Section: Audio-visual Speech Recognitionmentioning

confidence: 99%

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

Bhaskar

Madathil

2022

IJEEI

View full text Add to dashboard Cite

Multimodal Integration (MI) is the study of merging the knowledge acquired by the nervous system using sensory modalities such as speech, vision, touch, and gesture. The applications of MI expand over the areas of Audio-Visual Speech Recognition (AVSR), Sign Language Recognition (SLR), Emotion Recognition (ER), Bio Metrics Applications (BMA), Affect Recognition (AR), Multimedia Retrieval (MR), etc. The fusion of modalities such as hand gestures-facial, lip-hand position, etc., are mainly used as sensory modalities to develop hard-of-hearing multimodal systems. This paper encapsulates an overview of multimodal systems available within the literature for hard-ofhearing studies. This paper also discusses some of the studies related to hardof-hearing acoustic analysis. It is observed that very few algorithms have been developed for hard-of-hearing AVSR than normal hearing. Thus, the study of audio-visual-based speech recognition systems for hard-of-hearing is highly demanded by people trying to communicate with natively speaking languages. This paper also highlights the state-of-the-art techniques in AVSR and the challenges faced by the researchers in the development of AVSR systems.

show abstract

“…Cooperation and collaboration is another important application of robotics (Naik, 2020;Palaz, Magimai-Doss, Collobert, 2019;Sharma, 2019]. The paper focuses on another dimension of robotics application -speech recognition for care and support.…”

Section: Related Workmentioning

confidence: 99%

HMM-based phoneme speech recognition system for the control and command of industrial robots

Naik

2021

Technical Transactions

View full text Add to dashboard Cite

In recent years, the integration of human-robot interaction with speech recognition has gained a lot of pace in the manufacturing industries. Conventional methods to control the robots include semi-autonomous, fully-autonomous, and wired methods. Operating through a teaching pendant or a joystick is easy to implement but is not effective when the robot is deployed to perform complex repetitive tasks. Speech and touch are natural ways of communicating for humans and speech recognition, being the best option, is a heavily researched technology. In this study, we aim at developing a stable and robust speech recognition system to allow humans to communicate with machines (roboticarm) in a seamless manner. This paper investigates the potential of the linear predictive coding technique to develop a stable and robust HMM-based phoneme speech recognition system for applications in robotics. Our system is divided into three segments: a microphone array, a voice module, and a robotic arm with three degrees of freedom (DOF). To validate our approach, we performed experiments with simple and complex sentences for various robotic activities such as manipulating a cube and pick and place tasks. Moreover, we also analyzed the test results to rectify problems including accuracy and recognition score.

show abstract

Visual Speech Recognition Using Optical Flow and Hidden Markov Model

Cited by 16 publications

References 19 publications

Deep Learning-Based Approach for Arabic Visual Speech Recognition

Deep Learning-Based Approach for Arabic Visual Speech Recognition

Multimodal Based Audio-Visual Speech Recognition for Hard-of-Hearing: State of the Art Techniques and Challenges

HMM-based phoneme speech recognition system for the control and command of industrial robots

Contact Info

Product

Resources

About