AVSE Challenge: Audio-Visual Speech Enhancement Challenge

Abstract

Audio-visual speech enhancement is the task of improving the quality of a speech signal when video of the speaker is available. It opens-up the opportunity of improving speech intelligibility in adverse listening scenarios that are currently too challenging for audio-only speech enhancement models. The Audio-Visual Speech Enhancement (AVSE) challenge aims to set the first benchmark in this area. We provide participants with datasets and scripts to test their audio-visual speech enhancement models under a common framework for both training and evaluation. The data is derived from real-world videos, and comprises noisy mixes, in which audio from target speaker is mixed with either a competing speaker or a noise signal. The submitted systems are evaluated by conducting AV intelligibility tests involving human participants. We expect this challenge to be a platform for advancing the field of audio-visual speech-enhancement and to provide further insight about the scope and limitations of current AV speech enhancement approaches.

Efficient Intelligibility Evaluation Using Keyword Spotting: A Study on Audio-Visual Speech Enhancement

“…To create the AV dataset used in the evaluation, we selected a set of TED and TEDx videos 1 of public lectures delivered by a single speaker. Details about the train, dev and eval AV datasets can be found in [14]. After selecting the videos, we extracted sentences based on the manual transcriptions of the talks.…”

“…After validating our proposed method we conducted a largescale evaluation of speech enhancement systems submitted to [14]. We evaluated nine systems (including the baseline model), and the original (i.e., not enhanced) samples.…”

“…To design this closed set, choose target words and sentences we mine pre-existing stimuli using a phonetic dictionary and a language model. To validate the method we present details of the first wide scale evaluation of audio-visual speech enhancement systems, performed for the AVSE 2022 Challenge [14]. We show how the proposed paradigm resembles a transcription-based task in terms of ranking intelligibility of stimuli mixed with different maskers at different levels and that the new method is suitable for ranking the performance of AVSE systems,…”

Socio-Technical Trust For Multi-Modal Hearing Assistive Technology

“…Integrating them seamlessly can be a significant challenge to achieve a comprehensive and effective AV HAT technology for individuals with hearing loss. The new Audio-Visual Speech Enhancement (AVSE) Challenge takes the first step toward accomplishing this by setting benchmarks in this research area [9].…”

“…1 https://www.who.int/news-room/fact-sheets/deta il/deafness-and-hearing-loss processing [7]. The multi-modal aspect of AV hearing assistive technology (HAT) may provide a range of benefits for users, including the capability to selectively enhance speech based on the user's eye gaze [8,9] and lipreading-based technologies [10]. Given that speech enhancement in noisy environments is especially challenging, adding the visual aspect to hearing aid algorithms has been predicted to result in more reliable performance [11].…”