Deep Learning-Based Approach for Arabic Visual Speech Recognition

Abstract

Lip-reading technologies are rapidly progressing following the breakthrough of deep learning. It plays a vital role in its many applications, such as: human-machine communication practices or security applications. In this paper, we propose to develop an effective lip-reading recognition model for Arabic visual speech recognition by implementing deep learning algorithms. The Arabic visual datasets that have been collected contains 2400 records of Arabic digits and 960 records of Arabic phrases from 24 native speakers. The primary purpose is to provide a high-performance model in terms of enhancing the preprocessing phase. Firstly, we extract keyframes from our dataset. Secondly, we produce a Concatenated Frame Images (CFIs) that represent the utterance sequence in one single image. Finally, the VGG-19 is employed for visual features extraction in our proposed model. We have examined different keyframes: 10, 15, and 20 for comparing two types of approaches in the proposed model: (1) the VGG-19 base model and (2) VGG-19 base model with batch normalization. The results show that the second approach achieves greater accuracy: 94% for digit recognition, 97% for phrase recognition, and 93% for digits and phrases recognition in the test dataset. Therefore, our proposed model is superior to models based on CFIs input.

Lip Movement Recognition using Histogram of Oriented Gradient (HOG) and Support Machine Vector (SVM) for Arabic Word

“…The research includes important processes such as face detection, lip localization, feature extraction, classifier training, and word recognition. The accuracy of digit recognition was 94%, phrase recognition was 97%, and phrase and digit recognition was 93% [10].…”

Visual Lip-Reading for Quranic Arabic Alphabets and Words Using Deep Learning

“…For testing their proposed system, they used nine sentences uttered by 9 Arabic speakers and achieved 62.9% accuracy. Recently, researchers continued to improve the recognition process and performance at this level, as shown in [26]; they presented an Arabic dataset of sentences and numbers for visual speech recognition purposes, which contains 960 sentence videos and 2400 number videos from 24 speakers. They used concatenated frame images (CFIs) as preprocessing for their dataset, resulting in one single image containing utterance sequences, which are fed into their proposed model: visual geometry group (VGG-19) network with batch normalization for feature extraction and classification.…”

A Review on Intelligent Control Theory and Applications in Process Optimization and Smart Manufacturing

“…Beyond robotics, deep learning finds applications in diverse domains including image analysis [156][157][158], video analysis [159][160][161], natural language generation [162,163], speech recognition [164][165][166], biometrics [167,168], text analytics [169,170], and natural language processing [171][172][173].…”