MyLipper: A Personalized System for Speech Reconstruction using Multi-view Visual Feeds

Kumar, Yaman; Jain, Rohit; Salik, Mohd; Shah, Rajiv Ratn; Zimmermann, Roger; Yin, Yifang

doi:10.1109/ism.2018.00-19

Cited by 21 publications

(28 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This method is extended in [9] by adding optical flow information as input to the network and by adding a postprocessing step, where generated sound features are replaced by their closest match from the training set. A similar method that uses multi-view visual feeds has been proposed in [10]. Finally, Akbari et.…”

Section: Introductionmentioning

confidence: 99%

Video-Driven Speech Reconstruction Using Generative Adversarial Networks

Vougioukas

Petridis

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

Speech is a means of communication which relies on both audio and visual information. The absence of one modality can often lead to confusion or misinterpretation of information. In this paper we present an end-to-end temporal model capable of directly synthesising audio from silent video, without needing to transform to-and-from intermediate features. Our proposed approach, based on GANs is capable of producing natural sounding, intelligible speech which is synchronised with the video. The performance of our model is evaluated on the GRID dataset for both speaker dependent and speaker independent scenarios. To the best of our knowledge this is the first method that maps video directly to raw audio and the first to produce intelligible speech when tested on previously unseen speakers. We evaluate the synthesised audio not only based on the sound quality but also on the accuracy of the spoken words.

show abstract

Section: Introductionmentioning

confidence: 99%

Video-Driven Speech Reconstruction Using Generative Adversarial Networks

Vougioukas

Petridis

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

show abstract

“…Despite much research in lipreading domain, it is still seen as a classification task in which, given some silent videos, a model has to classify those videos into a limited and fixed size of lexicon (Lucey and Potamianos 2006;Ngiam et al 2011;Lee, Lee, and Kim 2016;Zimmermann et al 2016;Assael et al 2016;Chung et al 2016;Petridis et al 2017; Chung and Zisserman 2017; Shah and Zimmermann 2017). There have also been a few works on speech-reconstruction as well (Cornu and Milner 2015;Kumar et al 2018a;2018b). However, the problem of view and pose-variation has been dealt by a very few lipreading systems (Zhou et al 2014).…”

Section: Related Workmentioning

confidence: 99%

“…One of such datasets is Oulu-VS2 (Anina et al 2015) which provides five different views of speakers shot concurrently. On this dataset, combination of multiple poses was tried for speechreading by (Petridis et al 2017) and for speech-reconstruction by (Kumar et al 2018a;2018b). Given visual feeds from multiple cameras, the authors showed that combining multiple views would result in better accuracy in speechreading (Lucey and Potamianos 2006;Zimmermann et al 2016;Lee, Lee, and Kim 2016;Petridis et al 2017) and better speech quality (Kumar et al 2018a;2018b) for speech-reconstruction.…”

Section: Related Workmentioning

confidence: 99%

“…We choose Perceptual Evaluation of Speech Quality (PESQ) (Rix et al 2001) as the evaluation metric to judge the quality of the sound. This metric has been used by other speechreconstruction works as well Ephrat, Halperin, and Peleg 2017;Kumar et al 2018a;2018b). PESQ is a ITU-T recommended standard for evaluating speech quality of 3.2 kHz codecs (Recommendation 2001).…”

Section: Evaluation Metricmentioning

confidence: 99%

“…Ong and Bowden 2011;Zhou, Zhao, and Pietikäinen 2011;Pei, Kim, and Zha 2013;Rekik, Ben-Hamadou, and Mahdi 2014;Petridis et al 2017;Kumar et al 2018a;2018b).…”

unclassified

See 2 more Smart Citations

Lipper: Synthesizing Thy Speech Using Multi-View Lipreading

Kumar

Jain

Salik

et al. 2019

AAAI

Self Cite

View full text Add to dashboard Cite

Lipreading has a lot of potential applications such as in the domain of surveillance and video conferencing. Despite this, most of the work in building lipreading systems has been limited to classifying silent videos into classes representing text phrases. However, there are multiple problems associated with making lipreading a text-based classification task like its dependence on a particular language and vocabulary mapping. Thus, in this paper we propose a multi-view lipreading to audio system, namely Lipper, which models it as a regression task. The model takes silent videos as input and produces speech as the output. With multi-view silent videos, we observe an improvement over single-view speech reconstruction results. We show this by presenting an exhaustive set of experiments for speaker-dependent, out-of-vocabulary and speaker-independent settings. Further, we compare the delay values of Lipper with other speechreading systems in order to show the real-time nature of audio produced. We also perform a user study for the audios produced in order to understand the level of comprehensibility of audios produced using Lipper.

show abstract