Survey on automatic lip-reading in the era of deep learning

Abstract

In the last few years, there has been an increasing interest in developing systems for Automatic LipReading (ALR). Similarly to other computer vision applications, methods based on Deep Learning (DL) have become very popular and have permitted to substantially push forward the achievable performance. In this survey, we review ALR research during the last decade, highlighting the progression from approaches previous to DL (which we refer to as traditional) toward end-to-end DL architectures. We provide a comprehensive list of the audiovisual databases available for lipreading, describing what tasks they can be used for, their popularity and their most important characteristics, such as the number of speakers, vocabulary size, recording settings and total duration. In correspondence with the shift toward DL, we show that there is a clear tendency toward large-scale datasets targeting realistic application settings and large numbers of samples per class. On the other hand, we summarize, discuss and compare the different ALR systems proposed in the last decade, separately considering traditional and DL approaches. We address a quantitative analysis of the different systems by organizing them in terms of the task that they target (e.g. recognition of letters or digits and words or sentences) and comparing their reported performance in the most commonly used datasets. As a result, we find that DL architectures perform similarly to traditional ones for simpler tasks but report significant improvements in more complex tasks, such as word or sentence recognition, with up to 40% improvement in word recognition rates. Hence, we provide a detailed description of the available ALR systems based on end-to-end DL architectures and identify a tendency to focus on the modeling of temporal context as the key to advance the field. Such modeling is dominated by recurrent neural networks due to their ability to retain context at multiple scales (e.g. short-and longterm information). In this sense, current efforts tend toward techniques that allow a more comprehensive modeling and interpretability of the retained context.

LSTM Based Lip Reading Approach for Devanagiri Script

“…In such environments, there is a need to build a speech recognizer that can use both audio and facial visual information. It would also be necessary to have a speech recognizer that would solely use visual information and does not depend on audio input (Fernandez-Lopez & Sukno, 2018). i.e., such speech recognizer should extract features from the facial parts, especially lip movement, and facial expressions.…”

“…i.e., such speech recognizer should extract features from the facial parts, especially lip movement, and facial expressions. Several attempts have been made to detect speech from the lip movement by (Almajai, Cox, Harvey, & Lan, 2016;Chung, Senior, Vinyals, & Zisserman, 2017;Dupont & Luettin, 2000;Petridis & Pantic, 2016;Sui, Bennamoun, & Togneri, 2015;Wand, Koutnik, & Schmidhuber, 2016;Yau, Kumar, & Weghorn, 2007;Zhou, Zhao, Hong, & Pietikäinen, 2014), but the results are much low as compared with audio speech recognizers (Fernandez-Lopez & Sukno, 2018).…”

“…But some of the authors argue that a sequence of words appearing in a sentence has a pattern and such sequence of words can resolve the mapping of lip movement patterns to a word (Afouras, Chung, & Zisserman, 2018;Assael, Shillingford, Whiteson, & de Freitas, 2016;. Therefore, the autonomous lip-reading system with high accuracy remains to be challenging one (Fernandez-Lopez & Sukno, 2018 poor video resolution and low frame rates of videos (Buchan, Paré, & Munhall, 2007;Hilder et al, 2009;Ortiz, 2008).…”

“…The recent works have used Recurrent neural network/ LSTM to do the same. When their results are compared, the Markova modules perform better than LSTM, and it is mainly due to the unavailability of large data repositories (Fernandez-Lopez & Sukno, 2018). All the earlier research works on the lip-reading system have three steps in common (Fernandez-Lopez & Sukno, 2018).…”

MobiLipNet: Resource-Efficient Deep Learning Based Lipreading

“…The field of visual speech recognition (VSR), or lipreading, has witnessed dramatic breakthroughs recently, primarily due to the paradigm shift from hand-crafted features to deep learning based models [1][2][3][4][5][6][7][8], coupled with the public release of large suitable corpora in a variety of environments [9][10][11][12][13][14][15], as also reviewed in [16,17]. Such models however, while reducing recognition errors compared to previous approaches, are not as efficient to compute and store.…”