Vid2speech: Speech reconstruction from silent video

Ephrat, Ariel; Peleg, Bezalel

doi:10.1109/icassp.2017.7953127

Cited by 108 publications

(115 citation statements)

References 16 publications

Supporting

Mentioning

114

Contrasting

Order By: Relevance

“…In this experiment, we attempted to separate the speech of two 'unknown' speakers. First, we trained a vid2speech network [5] on the data of a 'known' speaker (S2 from GRID). The training data consisted of randomly selected sentences (40 minutes length in total).…”

Section: Resultsmentioning

confidence: 99%

“…Several approaches exist for generation of intelligible speech from silent video frames of a person speaking [5,6,7]. In this work we rely on vid2speech [6], briefly described in Sec.…”

Section: Visually-derived Speech Generationmentioning

confidence: 99%

“…We continue with the isolation of the speech of a single visible speaker from background sounds. This work builds upon recent advances in machine speechreading, generating speech from visible motion of the face and mouth [5,6,7].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Seeing Through Noise: Visually Driven Speaker Separation And Enhancement

Gabbay

Ephrat

Halperin

et al. 2018

2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Isolating the voice of a specific person while filtering out other voices or background noises is challenging when video is shot in noisy environments. We propose audio-visual methods to isolate the voice of a single speaker and eliminate unrelated sounds. First, face motions captured in the video are used to estimate the speaker's voice, by passing the silent video frames through a video-to-speech neural network-based model. Then the speech predictions are applied as a filter on the noisy input audio. This approach avoids using mixtures of sounds in the learning process, as the number of such possible mixtures is huge, and would inevitably bias the trained model. We evaluate our method on two audio-visual datasets, GRID and TCD-TIMIT, and show that our method attains significant SDR and PESQ improvements over the raw video-to-speech predictions, and a well-known audio-only method.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Several approaches exist for generation of intelligible speech from silent video frames of a person speaking [5,6,7]. In this work we rely on vid2speech [6], briefly described in Sec.…”

Section: Visually-derived Speech Generationmentioning

confidence: 99%

See 1 more Smart Citation

Seeing Through Noise: Visually Driven Speaker Separation And Enhancement

Gabbay

Ephrat

Halperin

et al. 2018

2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Convolutional neural networks (CNNs) have been shown to be powerful feature extractors for images and videos and have replaced handcrafted features in more recent works. One such system is proposed in [8] to predict line spectrum pairs (LSPs) from video. The LSPs are converted into waveforms but since excitation is not predicted the resulting speech sounds unnatural.…”

Section: Introductionmentioning

confidence: 99%

Video-Driven Speech Reconstruction Using Generative Adversarial Networks

Vougioukas

Petridis

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

Speech is a means of communication which relies on both audio and visual information. The absence of one modality can often lead to confusion or misinterpretation of information. In this paper we present an end-to-end temporal model capable of directly synthesising audio from silent video, without needing to transform to-and-from intermediate features. Our proposed approach, based on GANs is capable of producing natural sounding, intelligible speech which is synchronised with the video. The performance of our model is evaluated on the GRID dataset for both speaker dependent and speaker independent scenarios. To the best of our knowledge this is the first method that maps video directly to raw audio and the first to produce intelligible speech when tested on previously unseen speakers. We evaluate the synthesised audio not only based on the sound quality but also on the accuracy of the spoken words.

show abstract

“…This approach had the limitation of missing certain speech components such as fundamental frequency and aperiodicity which was then determined artificially thereby compromising quality in order to maximize intelligibility. Ephrat and Peleg (2017) modified this technique by using an end-to-end CNN to extract visual features from the entire face while applying a similar approach for modeling audio features using 8th order Linear Predictive Coding (LPC) analysis followed by Line Spectrum Pairs (LSP) decomposition. However, it also suffered from the same missing excitation parameters resulting in an unnatural sounding voice.…”

Section: Related Workmentioning

confidence: 99%

Lip2Audspec: Speech Reconstruction from Silent Lip Movements Video

Akbari

Arora

Cao

et al. 2018

2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this study, we propose a deep neural network for reconstructing intelligible speech from silent lip movement videos. We use auditory spectrogram as spectral representation of speech and its corresponding sound generation method resulting in a more natural sounding reconstructed speech. Our proposed network consists of an autoencoder to extract bottleneck features from the auditory spectrogram which is then used as target to our main lip reading network comprising of CNN, LSTM and fully connected layers. Our experiments show that the autoencoder is able to reconstruct the original auditory spectrogram with a 98% correlation and also improves the quality of reconstructed speech from the main lip reading network. Our model, trained jointly on different speakers is able to extract individual speaker characteristics and gives promising results of reconstructing intelligible speech with superior word recognition accuracy.

show abstract

Vid2speech: Speech reconstruction from silent video

Cited by 108 publications

References 16 publications

Seeing Through Noise: Visually Driven Speaker Separation And Enhancement

Seeing Through Noise: Visually Driven Speaker Separation And Enhancement

Video-Driven Speech Reconstruction Using Generative Adversarial Networks

Lip2Audspec: Speech Reconstruction from Silent Lip Movements Video

Contact Info

Product

Resources

About