VoViT: Low Latency Graph-Based Audio-Visual Voice Separation Transformer

Montesinos, Juan F.; Kadandale, Venkatesh S.; Haro, Gloria

doi:10.1007/978-3-031-19836-6_18

Cited by 8 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dataset. We use Acappella [10], the only publicly available AV singing voice dataset. It consists of around 46-hours of solosinging videos spanning four language categories.…”

Section: Lip Synchronisation In Singing Voicementioning

confidence: 99%

“…Training. We use the same training pipeline that is used for training the model Y-Net-mr in [10]. Y-Net-mr is a U-Net [23] conditioned by visual features extracted from cropped mouth frames using an 18-layer mixed convolution network [22].…”

Section: Singing Voice Separationmentioning

confidence: 99%

“…Models. The main baseline model here is the Y-Net-mr from [10] which has been trained end-to-end for singing voice separation. It is a state-of-the-art audio-visual singing voice separation model among the models that operate directly on the cropped mouth frames.…”

Section: Singing Voice Separationmentioning

confidence: 99%

“…Our lip synchronisation detector outperforms state-ofthe-art models on the speech benchmark dataset Lip Reading Sentences 2 (LRS2) [9]. Besides, we also train and test our model on the Acappella [10] dataset which contains videos of solo singing performances. We are the first to explore the relevance of lip-sync detectors trained in speech videos to detect lip-sync in the singing voice.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices

Kadandale¹,

Montesinos²,

Haro³

2022

Interspeech 2022

View full text Add to dashboard Cite

In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audiovisual correspondence score. We propose an audio-visual crossmodal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the singing voice. The singing voice is a more challenging use case for synchronisation due to sustained vowel sounds. We also investigate the relevance of lip synchronisation models trained on speech datasets in the context of singing voice. Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end. The demos, source code, and the pre-trained models are available on https://ipcv.github.io/VocaLiST/

show abstract

“…Dataset. We use Acappella [10], the only publicly available AV singing voice dataset. It consists of around 46-hours of solosinging videos spanning four language categories.…”

Section: Lip Synchronisation In Singing Voicementioning

confidence: 99%