Multi-view Automatic Lip-Reading Using Neural Network

Lee, Dae-Hyun; Lee, Jongmin; Kim, Kee-Eung

doi:10.1007/978-3-319-54427-4_22

Cited by 47 publications

(59 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most common confusion pair 2 for the OuluVS2 dataset is between "Hello" (3rd phrase) and "Thank you" (8th phrase) which is consistent with confusions presented in (Petridis et al, 2017a;Lee et al, 2016). The most frequently confused pairs in the CUAVE dataset are zero and two, and six and nine and this is consistent with (Petridis et al, 2017a).…”

Section: Methodssupporting

confidence: 75%

End-to-end visual speech recognition for small-scale datasets

Petridis

Wang

et al. 2020

Pattern Recognition Letters

View full text Add to dashboard Cite

Visual speech recognition models traditionally consist of two stages, feature extraction and classification. Several deep learning approaches have been recently presented aiming to replace the feature extraction stage by automatically extracting features from mouth images. However, research on simultaneously learning features and performing classification remains limited. In addition, most of the existing methods require large amounts of data in order to achieve state-of-the-art performance, otherwise they under-perform. In this work, an end-to-end visual speech recognition system is presented based on fully-connected layers and Long-Short Memory (LSTM) networks which is suitable for small-scale datasets. The model consists of two streams: one which extract features directly from the mouth images and one which extracts features from the difference images. A Bidirectional LSTM (BLSTM) is used for modelling the temporal dynamics in each stream which are then fused via another BLSTM. An absolute improvement of 0.6%, 3.4%, 3.9%, 11.4% over the state-of-the-art is reported on the OuluVS2, CUAVE, AVLetters and AVLetters2 databases, respectively.

show abstract

Section: Methodssupporting

confidence: 75%

End-to-end visual speech recognition for small-scale datasets

Petridis

Wang

et al. 2020

Pattern Recognition Letters

View full text Add to dashboard Cite

show abstract

“…The only combination which outperformed the frontal view is frontal + 30 • . Similarly to [19], the combination of all views led to worse performance than most individual views.…”

Section: Related Workmentioning

confidence: 97%

“…The frontal view was found to be the best by Saitoh et al [29], the profile view by Lee at al. [19], the 30 • view by Zimmermann et al [34] and the 60 • view was found to be the best performing in Three different convolutional neural networks (CNNs), GoogLeNet, AlexNet and Network in Network, were trained on OuluVS2 using data augmentation. Each model led to different performance across the views.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

End-to-End Multi-View Lipreading

Petridis¹,

Wang²,

Li³

et al. 2017

Procedings of the British Machine Vision Conference 2017

View full text Add to dashboard Cite

Non-frontal lip views contain useful information which can be used to enhance the performance of frontal view lipreading. However, the vast majority of recent lipreading works, including the deep learning approaches which significantly outperform traditional approaches, have focused on frontal mouth images. As a consequence, research on joint learning of visual features and speech classification from multiple views is limited. In this work, we present an end-to-end multi-view lipreading system based on Bidirectional Long-Short Memory (BLSTM) networks. To the best of our knowledge, this is the first model which simultaneously learns to extract features directly from the pixels and performs visual speech classification from multiple views and also achieves state-of-the-art performance. The model consists of multiple identical streams, one for each view, which extract features directly from different poses of mouth images. The temporal dynamics in each stream/view are modelled by a BLSTM and the fusion of multiple streams/views takes place via another BLSTM. An absolute average improvement of 3% and 3.8% over the frontal view performance is reported on the OuluVS2 database when the best two (frontal and profile) and three views (frontal, profile, 45 • ) are combined, respectively. The best three-view model results in a 10.5% absolute improvement over the current multi-view state-of-the-art performance on OuluVS2, without using external databases for training, achieving a maximum classification accuracy of 96.9%.

show abstract

“…The final individual weights are a combination of these coarser weights. Finally, some recent research has applied cross-view analysis to 3D-AAMs [30] and used channel, image and feature fusion for multiple-and cross-view analysis [31].…”

Section: Related Workmentioning

confidence: 99%

Combining Multiple Views for Visual Speech Recognition

Zimmermann¹,

Ghazi²,

Ekenel³

et al. 2017

The 14th International Conference on Auditory-Visual Speech Processing

View full text Add to dashboard Cite

Visual speech recognition is a challenging research problem with a particular practical application of aiding audio speech recognition in noisy scenarios. Multiple camera setups can be beneficial for the visual speech recognition systems in terms of improved performance and robustness. In this paper, we explore this aspect and provide a comprehensive study on combining multiple views for visual speech recognition. The thorough analysis covers fusion of all possible view angle combinations both at feature level and decision level. The employed visual speech recognition system in this study extracts features through a PCA-based convolutional neural network, followed by an LSTM network. Finally, these features are processed in a tandem system, being fed into a GMM-HMM scheme. The decision fusion acts after this point by combining the Viterbi path log-likelihoods. The results show that the complementary information contained in recordings from different view angles improves the results significantly. For example, the sentence correctness on the test set is increased from 76% for the highest performing single view (30 • ) to up to 83% when combining this view with the frontal and 60 • view angles.

show abstract

Multi-view Automatic Lip-Reading Using Neural Network

Cited by 47 publications

References 18 publications

End-to-end visual speech recognition for small-scale datasets

End-to-end visual speech recognition for small-scale datasets

End-to-End Multi-View Lipreading

Combining Multiple Views for Visual Speech Recognition

Contact Info

Product

Resources

About