Advances and Challenges in Deep Lip Reading

Oghbaie, Marzieh; Sabaghi, Arian; Kooshan, Hashemifard,; Akbari, Mohammad Esmaeil

doi:10.48550/arxiv.2110.07879

Cited by 2 publications

(2 citation statements)

References 101 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, in the CTC configuration, 'Hel-lo' is the correct representation of 'Hello,' where 'l' is duplicated. The CTC loss function accepts a model output matrix consisting of scores assigned to each token at every time step alongside the actual truth sequence [27].…”

Section: Ctc Loss Functionmentioning

confidence: 99%

Lip-Reading with Visual Form Classification using Residual Networks and Bidirectional Gated Recurrent Units

Anni,

Suharjito

2023

HighTech. Innov. J.

View full text Add to dashboard Cite

Lip-reading is a method that focuses on the observation and interpretation of lip movements to understand spoken language. Previous studies have exclusively concentrated on a single variation of residual networks (ResNets). This study primarily aimed to conduct a comparative analysis of several types of ResNets. This study additionally calculates metrics for several word structures included in the GRID dataset, encompassing verbs, colors, prepositions, letters, and numerals. This component has not been previously investigated in other studies. The proposed approach encompasses several stages, namely pre-processing, which involves face detection and mouth location, feature extraction, and classification. The architecture for feature extraction comprises a 3-dimensional convolutional neural network (3D-CNN) integrated with ResNets. The management of temporal sequences during the classification phase is accomplished through the utilization of the bidirectional gated recurrent units (Bi-GRU) model. The experimental results demonstrated a character error rate (CER) of 14.09% and a word error rate (WER) of 28.51%. The combination of 3D-CNN ResNet-34 and Bi-GRU yielded superior outcomes in comparison to ResNet-18 and ResNet-50. The correlation between increased network depth and enhanced performance in lip-reading models was not consistently observed. Nevertheless, the incorporation of additional trained parameters offers certain benefits. Moreover, it has demonstrated superior levels of precision in comparison to human professionals in the task of distinguishing diverse word structures. Doi: 10.28991/HIJ-2023-04-02-010 Full Text: PDF

show abstract

Section: Ctc Loss Functionmentioning

confidence: 99%

Lip-Reading with Visual Form Classification using Residual Networks and Bidirectional Gated Recurrent Units

Anni,

Suharjito

2023

HighTech. Innov. J.

View full text Add to dashboard Cite

show abstract

“…Sooraj et al conducted a review of the general structure of ALR systems and how they work [38]. Oghbaie similarly reviewed the general structure of ALR systems and determined areas for future advancements in the systems' structures [39]. Agrawal et al conducted a literature review that focused on the methods by which the networks are trained, particularly comparing the prepossessing and pretraining methods that benefit ALR systems [40].…”

Section: Introductionmentioning

confidence: 99%

Data-Driven Advancements in Lip Motion Analysis: A Review

Torrie,

Sumsion,

Lee

et al. 2023

Electronics

View full text Add to dashboard Cite

This work reviews the dataset-driven advancements that have occurred in the area of lip motion analysis, particularly visual lip-reading and visual lip motion authentication, in the deep learning era. We provide an analysis of datasets and their usage, creation, and associated challenges. Future research can utilize this work as a guide for selecting appropriate datasets and as a source of insights for creating new and innovative datasets. Large and varied datasets are vital to a successful deep learning system. There have been many incredible advancements made in these fields due to larger datasets. There are indications that even larger, more varied datasets would result in further improvement upon existing systems. We highlight the datasets that brought about the progression in lip-reading systems from digit- to word-level lip-reading, and then from word- to sentence-level lip-reading. Through an in-depth analysis of lip-reading system results, we show that datasets with large amounts of diversity increase results immensely. We then discuss the next step for lip-reading systems to move from sentence- to dialogue-level lip-reading and emphasize that new datasets are required to make this transition possible. We then explore lip motion authentication datasets. While lip motion authentication has been well researched, it is not very unified on a particular implementation, and there is no benchmark dataset to compare the various methods. As was seen in the lip-reading analysis, large, diverse datasets are required to evaluate the robustness and accuracy of new methods attempted by researchers. These large datasets have pushed the work in the visual lip-reading realm. Due to the lack of large, diverse, and publicly accessible datasets, visual lip motion authentication research has struggled to validate results and real-world applications. A new benchmark dataset is required to unify the studies in this area such that they can be compared to previous methods as well as validate new methods more effectively.

show abstract

Advances and Challenges in Deep Lip Reading

Cited by 2 publications

References 101 publications

Lip-Reading with Visual Form Classification using Residual Networks and Bidirectional Gated Recurrent Units

Lip-Reading with Visual Form Classification using Residual Networks and Bidirectional Gated Recurrent Units

Data-Driven Advancements in Lip Motion Analysis: A Review

Contact Info

Product

Resources

About