Hush-Hush Speak: Speech Reconstruction Using Silent Videos

Uttam, Shashwat; Kumar, Yaman; Sahrawat, Dhruva; Aggarwal, Mansi; Shah, Rajiv Ratn; Mahata, Debanjan; Stent, Amanda

doi:10.21437/interspeech.2019-3269

Cited by 14 publications

(10 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, AP and F0 were not estimated from the silent video, but artificially produced without taking the visual information into account, while SP was estimated with a Gaussian mixture model (GMM) and FFNN within a regression-based framework. As input to the models, two different visual features were considered, 2-D DCT and AAM, while the explored SP representations were [149] 2017 AAM Codebook entries FFNN / RNN mouth (mel-filterbank amplitudes) [57] 2017 Raw pixels LSP of LPC CNN, FFNN face [56] 2017 Raw pixels, Mel-scale and CNN, FFNN, optical flow linear-scale BiGRU face spectrograms [11] 2018 Raw pixels AE features, CNN, LSTM, face spectrogram FFNN, AE [145] 2018 Raw pixels LSP of LPC CNN, LSTM, mouth FFNN [147] 2018 Raw pixels LSP of LPC CNN, BiGRU, mouth FFNN [146] 2019 Raw pixels LSP of LPC CNN, BiGRU, mouth FFNN [243] 2019 Raw pixels WORLD CNN, FFNN mouth spectrum [256] 2019 Raw pixels Raw waveform GAN, CNN, mouth GRU [247] 2019 Raw pixels AE features, CNN, LSTM mouth spectrogram FFNN, AE [177] 2020 Raw pixels WORLD CNN, GRU, mouth / face features FFNN [206] 2020 Raw pixels mel-scale CNN, LSTM face spectrogram linear predictive coding (LPC) coefficients and mel-filterbank amplitudes. While the choice of visual features did not have a big impact on the results, the use of mel-filterbank amplitudes allowed to outperform the systems based on LPC coefficients.…”

Section: A Speech Reconstruction From Silent Videosmentioning

confidence: 99%

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Michelsanti

Tan

Zhang

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

188

View full text Add to dashboard Cite

Speech enhancement and speech separation are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. More recently, visual information from the target speakers, such as lip movements and facial expressions, has been introduced to speech enhancement and speech separation systems, because the visual aspect of speech is essentially unaffected by the acoustic environment. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically deep learning, achieving state-of-the-art performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: visual features; acoustic features; deep learning methods; fusion techniques; training targets and objective functions. We also survey commonly employed audio-visual speech datasets, given their central role in the development of data-driven approaches, and evaluation methods, because they are generally used to compare different systems and determine their performance. In addition, we review deeplearning-based methods for speech reconstruction from silent videos and audio-visual sound source separation for non-speech signals, since these methods can be more or less directly applied to audio-visual speech enhancement and separation.

show abstract

Section: A Speech Reconstruction From Silent Videosmentioning

confidence: 99%

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Michelsanti

Tan

Zhang

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

188

View full text Add to dashboard Cite

show abstract

“…With advancements in deep learning and the increasing availability of large lip reading datasets, there have been approaches such as (Wand, Koutník, and Schmidhuber 2016;Zhou et al 2019;Salik et al 2019) addressing lip reading using deep learning based algorithms such as CNNs and LSTMs. Further, there have been works such as (Lee, Lee, and Kim 2016;Kumar et al 2018b;Uttam et al 2019;Kumar et al 2019) extending lip reading from single view to multiview settings by incorporating videos of mouth sections from multiple views together. Multi-view lip reading has been shown to improve performance significantly as compared to single view.…”

Section: Visual Speech Recognitionmentioning

confidence: 99%

Harnessing GANs for Zero-Shot Learning of New Classes in Visual Speech Recognition

Kumar¹,

Sahrawat²,

Maheshwari³

et al. 2020

AAAI

Self Cite

View full text Add to dashboard Cite

Visual Speech Recognition (VSR) is the process of recognizing or interpreting speech by watching the lip movements of the speaker. Recent machine learning based approaches model VSR as a classification problem; however, the scarcity of training data leads to error-prone systems with very low accuracies in predicting unseen classes. To solve this problem, we present a novel approach to zero-shot learning by generating new classes using Generative Adversarial Networks (GANs), and show how the addition of unseen class samples increases the accuracy of a VSR system by a significant margin of 27% and allows it to handle speaker-independent out-of-vocabulary phrases. We also show that our models are language agnostic and therefore capable of seamlessly generating, using English training data, videos for a new language (Hindi). To the best of our knowledge, this is the first work to show empirical evidence of the use of GANs for generating training samples of unseen classes in the domain of VSR, hence facilitating zero-shot learning. We make the added videos for new classes publicly available along with our code1.

show abstract

“…With the advancements in deep learning and the increasing availability of large-scale lip reading datasets, approaches have been proposed that address lip reading using deep learning based algorithms (e.g., CNNs and LSTMs) [35,45,49]. Moreover, researchers have extended lip reading from single view to multi-view settings by incorporating videos of mouth section from multiple views together [21,22,24,37,44],. Multi-view lip reading has shown to improve performance significantly as compared to single view.…”

Section: Related Workmentioning

confidence: 99%

"Notic My Speech" -- Blending Speech Patterns With Multimedia

Sahrawat¹,

Kumar²,

Aggarwal³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Speech as a natural signal is composed of three parts -visemes (visual part of speech), phonemes (spoken part of speech), and language (the imposed structure). However, video as a medium for the delivery of speech and a multimedia construct has mostly ignored the cognitive aspects of speech delivery. For example, video applications like transcoding and compression have till now ignored the fact how speech is delivered and heard. To close the gap between speech understanding and multimedia video applications, in this paper, we show the initial experiments by modelling the perception on visual speech and showing its use case on video compression. On the other hand, in the visual speech recognition domain, existing studies have mostly modeled it as a classification problem, while ignoring the correlations between views, phonemes, visemes, and speech perception. This results in solutions which are further away from how human perception works. To bridge this gap, we propose a view-temporal attention mechanism to model both the view dependence and the visemic importance in speech recognition and understanding. We conduct experiments on three public visual speech recognition datasets. The experimental results show that our proposed method outperformed the existing work by 4.99% in terms of the viseme error rate. Moreover, we show that there is a strong correlation between our model's understanding of multi-view speech and the human perception. This characteristic benefits downstream applications such as video compression and streaming where a significant number of less important frames can be compressed or eliminated while being able to maximally preserve human speech understanding with good user experience.

show abstract

Hush-Hush Speak: Speech Reconstruction Using Silent Videos

Cited by 14 publications

References 12 publications

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Harnessing GANs for Zero-Shot Learning of New Classes in Visual Speech Recognition

"Notic My Speech" -- Blending Speech Patterns With Multimedia

Contact Info

Product

Resources

About