Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss

Makishima, Naoki; Ihori, Mana; Takashima, Akihiko; Tanaka, Tomohiro; Orihashi, Shota; Masumura, Ryo

doi:10.1109/icassp39728.2021.9413491

Cited by 7 publications

(5 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Makishima et al [21] proposed a model with two subnetworks for visual and auditory inputs. Embeddings are produced through different sub-networks, which are subsequently integrated and processed by a decoder network.…”

Section: Related Workmentioning

confidence: 99%

Audio-Visual Source Separation Based Fusion Techniques

Mudhafar,

Al Tmeme

2024

RIA

View full text Add to dashboard Cite

A novel hybrid deep learning model for audio-visual source separation is introduced in this paper, with a specific focus on the precise isolation of a particular speaker's voice from video content. By leveraging both audio and visual characteristics, the achievement of accurate separation of the targeted speech signal is facilitated by our model. Notably, the incorporation of the speaker's facial expressions as an auxiliary cue for enhancing the extraction of their unique vocal qualities is emphasized. Proficiency in audio-visual speech separation and latent representations of distinctive speaker attributes, known as speaker embeddings, is simultaneously acquired by our model through unsupervised learning on unannotated video data. The model employed in this study is speaker-independent, wherein an initial stage of feature extraction is conducted for both audio and visual inputs prior to the subsequent deep modal analysis. The utilization of facial attribute features as an identifying code enables the identification of the speaker's frequency space or other audio properties. The model's efficacy was assessed through evaluation on the widely recognized AVspeech dataset yielding an improvement of 7.7 in terms of source-to-distortion ratio (SDR).

show abstract

Section: Related Workmentioning

confidence: 99%

Audio-Visual Source Separation Based Fusion Techniques

Mudhafar,

Al Tmeme

2024

RIA

View full text Add to dashboard Cite

show abstract

“…Makishima et al [38] proposed a deep learning method that utilizes both auditory and visual information to separate speech signals from an audio-visual mixture. It consists of two subnetworks, a visual network and an auditory network, which generate embeddings that are concatenated and passed through a decoder network.…”

Section: Recent Avss Workmentioning

confidence: 99%

“…One advanced method for audio-visual source separation involves the use of deep learning techniques [10][11][12]14,22,34,[37][38].…”

Section: Introductionmentioning

confidence: 99%

An Overview of Audio-Visual Source Separation Using Deep Learning

Sulaiman,

Al Tmeme,

Mahdi

2023

alkej

View full text Add to dashboard Cite

In this article, the research presents a general overview of deep learning-based AVSS (audio-visual source separation) systems. AVSS has achieved exceptional results in a number of areas, including decreasing noise levels, boosting speech recognition, and improving audio quality. The advantages and disadvantages of each deep learning model are discussed throughout the research as it reviews various current experiments on AVSS. The TCD TIMIT dataset (which contains top-notch audio and video recordings created especially for speech recognition tasks) and the Voxceleb dataset (a sizable collection of brief audio-visual clips with human speech) are just a couple of the useful datasets summarized in the paper that can be used to test AVSS systems. In its basic form, this review aims to highlight the growing importance of AVSS in improving the quality of audio signals.

show abstract

“…Most recent algorithms made use of lips motion as well as appearance information, usually implementing cross-modal losses to pull together corresponding audio-visual features Gao and Grauman [2021], Makishima et al [2021].…”

Section: Related Workmentioning

confidence: 99%

VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer

Montesinos¹,

Kadandale²,

Haro³

2022

Preprint

View full text Add to dashboard Cite

This paper presents an audio-visual approach for voice separation which outperforms state-of-theart methods at a low latency in two scenarios: speech and singing voice. The model is based on a two-stage network. Motion cues are obtained with a lightweight graph convolutional network that processes face landmarks. Then, both audio and motion features are fed to an audio-visual transformer which produces a fairly good estimation of the isolated target source. In a second stage, the predominant voice is enhanced with an audio-only network. We present different ablation studies and comparison to state-of-the-art methods. Finally, we explore the transferability of models trained for speech separation in the task of singing voice separation. The demos, code, and weights will be made publicly available at https://ipcv.github.io/VoViT/

show abstract

Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss

Cited by 7 publications

References 31 publications

Audio-Visual Source Separation Based Fusion Techniques

Audio-Visual Source Separation Based Fusion Techniques

An Overview of Audio-Visual Source Separation Using Deep Learning

VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer

Contact Info

Product

Resources

About