Audio-Visual Target Speaker Enhancement on Multi-Talker Environment using Event-Driven Cameras

Arriandiaga, Ander; Morrone, Giovanni; Pasa, Luca; Badino, Leonardo; Bartolozzi, Chiara

doi:10.48550/arxiv.1912.02671

Cited by 2 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Listening tests for speech DRT [255] 1983 Audio-only listening test using intelligibility assessment rhyming words HINT [191] 1994 Audio-only listening test using everyday sentences Matrix-like audio-visual 2019 Matrix test using audio-visual [178] test [178] stimuli [13] Estimators of speech quality PESQ [117], [119], [120], [214] 2001 Designed to assess quality across a [3], [5]- [7], [12], [17], [37], [55], [65] based on perceptual models wide range of codecs and network [66], [76], [77], [85], [99], [107], [108] conditions mostly for telephony [109], [122], [128], [136], [153], [154] [176], [178], [179], [183], [220]- [222] [239], [244], [263], [274], [279] CSIG / CBAK / COVRL [104] 2007 Composite measures which combine [108] basic objective measures HASQI [131], [133] 2010 Specifically designed for hearing- [99], [100] impaired listeners POLQA [1...…”

Section: Typementioning

confidence: 99%

“…[65] AAM of mouth region [136] 2D-DCT of mouth region [3]-[6] Optical flow [17], [65], [154], [164], [165] Landmark-based features [100], [154], [183], [203] Multisensory features [195] Face recognition embedding [55], [109], [169], [192], [239] VSR embedding [7], [10], [107]- [109], [153], [222], [273] Facial appearance embedding [42], [208] Compressed mouth frames [37] Speaker direction [85], [244], [279] Acoustic Features…”

Section: Audio-visual Speech Enhancement and Separation Systemsmentioning

confidence: 99%

See 1 more Smart Citation

Deep-learning-based audio-visual speech enhancement in presence of Lombard effect

Michelsanti

Sigurðsson

Jensen

2019

Speech Communication

View full text Add to dashboard Cite

Keywords:Lombard effect audio-visual speech enhancement deep learning speech quality speech intelligibility A B S T R A C T When speaking in presence of background noise, humans reflexively change their way of speaking in order to improve the intelligibility of their speech. This reflex is known as Lombard effect. Collecting speech in Lombard conditions is usually hard and costly. For this reason, speech enhancement systems are generally trained and evaluated on speech recorded in quiet to which noise is artificially added. Since these systems are often used in situations where Lombard speech occurs, in this work we perform an analysis of the impact that Lombard effect has on audio, visual and audio-visual speech enhancement, focusing on deep-learning-based systems, since they represent the current state of the art in the field.We conduct several experiments using an audio-visual Lombard speech corpus consisting of utterances spoken by 54 different talkers. The results show that training deep-learning-based models with Lombard speech is beneficial in terms of both estimated speech quality and estimated speech intelligibility at low signal to noise ratios, where the visual modality can play an important role in acoustically challenging situations. We also find that a performance difference between genders exists due to the distinct Lombard speech exhibited by males and females, and we analyse it in relation with acoustic and visual features. Furthermore, listening tests conducted with audio-visual stimuli show that the speech quality of the signals processed with systems trained using Lombard speech is statistically significantly better than the one obtained using systems trained with non-Lombard speech at a signal to noise ratio of −5 dB. Regarding speech intelligibility, we find a general tendency of the benefit in training the systems with Lombard speech.

show abstract

Section: Typementioning

confidence: 99%

Section: Audio-visual Speech Enhancement and Separation Systemsmentioning

confidence: 99%

Deep-learning-based audio-visual speech enhancement in presence of Lombard effect

Michelsanti

Sigurðsson

Jensen

2019

Speech Communication

View full text Add to dashboard Cite

show abstract

“…Estimators of speech quality SNR -It does not provide a proper [12], [65], [66], [109] based on energy ratios (Signal-to-Noise Ratio) estimation of speech distortion SSNR / SSNRI -Assessment of short-time [100], [108], [239] (Segmental SNR) behaviour (SSNR Improvement) SDI [31] 2006 It provides a rough distortion [99], [100] measure SDR [252] 2006 Specifically designed for blind audio [7], [10], [17], [42], [55], [65], [85] source separation [107]- [109], [136], [153], [154], [169] [164], [165], [183], [192], [195], [203] [208], [220]-[222] SIR [252] 2006 Specifically designed for blind audio [7], [65], [107], [136], [164], [165] source separation [195] SAR [252] 2006 Specifically designed for blind audio [65], [107], [136], [164], [165], [195] source separation SI-SDR [150]…”

Section: Ip Transmissionmentioning

confidence: 99%

“…Magnitude spectrogram [3]- [7], [10], [12], [17], [37], [42], [65], [66], [76], [77], [85], [99], [100], [107], [122], [128], [136], [153], [154], [164] [165], [176], [178], [179], [183], [192], [195], [203], [208], [220]- [222], [244], [263], [274], [279] Phase a…”

Section: Audio-visual Speech Enhancement and Separation Systemsmentioning

confidence: 99%

“…They have low latency, high dynamic range and very low power consumption [156]. Arriandiaga et al [17] showed that the SE results obtained with optical flow features, extracted from an event camera, are on par with a frame-based approach. The main limitation of exploiting the full potential of event cameras is that existing image processing algorithms cannot be employed, due to the inherently different nature of the data produced by them.…”

Section: A Visual Featuresmentioning

confidence: 99%

See 1 more Smart Citation

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Michelsanti

Tan

Zhang

et al. 2021

IEEE/ACM Trans. Audio Speech Lang. Process.

191

View full text Add to dashboard Cite

Speech enhancement and speech separation are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. More recently, visual information from the target speakers, such as lip movements and facial expressions, has been introduced to speech enhancement and speech separation systems, because the visual aspect of speech is essentially unaffected by the acoustic environment. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically deep learning, achieving state-of-the-art performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: visual features; acoustic features; deep learning methods; fusion techniques; training targets and objective functions. We also survey commonly employed audio-visual speech datasets, given their central role in the development of data-driven approaches, and evaluation methods, because they are generally used to compare different systems and determine their performance. In addition, we review deeplearning-based methods for speech reconstruction from silent videos and audio-visual sound source separation for non-speech signals, since these methods can be more or less directly applied to audio-visual speech enhancement and separation.

show abstract

Audio-Visual Target Speaker Enhancement on Multi-Talker Environment using Event-Driven Cameras

Cited by 2 publications

References 0 publications

Deep-learning-based audio-visual speech enhancement in presence of Lombard effect

Deep-learning-based audio-visual speech enhancement in presence of Lombard effect

An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

Contact Info

Product

Resources

About