End-to-End Active Speaker Detection

Alcázar, Juan León; Cordes, Moritz; Chen, Zhao; Ghanem, Bernard

doi:10.48550/arxiv.2203.14250

Cited by 2 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our extensive experiments demonstrate the effectiveness of our approach. On AVA-ActiveSpeaker dataset [36], Lo-CoNet achieves an mAP of 95.2%, outperforming current state-of-the-out method EASEE [4] by 1.1% despite using a simpler visual encoder. Furthermore, LoCoNet achieves 68.1% (+22%) on the Columbia dataset [10], 97.2% (+2.8%) on the Talkies dataset [32] and 59.7% (+8%) on the Ego4D dataset [19].…”

Section: Introductionmentioning

confidence: 93%

“…Types of encoder network. For the visual encoder, most ASD methods extract visual embeddings using 2D CNNs [20] since they require less GPU memory [3,30,32,38], although some work [4,30] use 3D CNNs [9]. For the audio encoder, existing audio backbones have high temporal downsampling which makes it hard to extract per-frame features [21,29].…”

Section: Active Speaker Detection (Asd)mentioning

confidence: 99%

“…One fundamental task is to identify, at any given moment, who is speaking in a complex scene with multiple interacting people. This Active Speaker Detection (ASD) problem [3,4,10,19,26,30,32,36,38] is important for many real-world applications, such as humanrobot interaction [24,37,39], speech diarization [14-16, 41, 44], video re-targeting [5,33,45] etc.…”

Section: Introductionmentioning

confidence: 99%

“…ASCNet [3] models long-term intra-and inter-speaker context. Other methods [4,30,32] model long-term intra-speaker and single-frame interspeaker context, but there is little dynamic inter-speaker contextual information within a single frame. TalkNet [38] long-term intra-speaker context modeling only models long-term intra-speaker context without considering other speakers.…”

Section: Introductionmentioning

confidence: 99%

“…Long-term intra-speaker context is modeled using a self-attention mechanism [40] due to its known effectiveness in modeling long-range dependencies, and interactions in short-term inter-speaker context are modeled with convolutional blocks that capture local patterns. In addition, while most ASD methods use vision backbones for audio encoding [3,4,32,38] due to high temporal downsampling in existing audio backbones, we propose VGGFrame, which is modified from VGGish [21] and can fully leverage pretrained AudioSet [17] weights to extract per-frame audio features.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

LoCoNet: Long-Short Context Network for Active Speaker Detection

Wang¹,

Cao²,

Bertasius³

et al. 2023

Preprint

View full text Add to dashboard Cite

Active Speaker Detection (ASD) aims to identify who is speaking in each frame of a video. ASD reasons from audio and visual information from two contexts: long-term intra-speaker context and short-term inter-speaker context. Long-term intra-speaker context models the temporal dependencies of the same speaker, while short-term inter-speaker context models the interactions of speakers in the same scene. These two contexts are complementary to each other and can help infer the active speaker. Motivated by these observations, we propose LoCoNet, a simple yet effective Long-Short Context Network that models the long-term intra-speaker context and short-term inter-speaker context. We use selfattention to model long-term intra-speaker context due to its effectiveness in modeling long-range dependencies, and convolutional blocks that capture local patterns to model short-term inter-speaker context. Extensive experiments show that LoCoNet achieves state-of-the-art performance on multiple datasets, achieving an mAP of 95.2%(+1.1%) on AVA-ActiveSpeaker, 68.1%(+22%) on Columbia dataset, 97.2%(+2.8%) on Talkies dataset and 59.7%(+8.0%) on Ego4D dataset. Moreover, in challenging cases where multiple speakers are present, or face of active speaker is much smaller than other faces in the same scene, LoCoNet outperforms previous state-of-the-art methods by 3.4% on the AVA-ActiveSpeaker dataset. The code will be released at:https: //github.com/SJTUwxz/LoCoNet_ASD.

show abstract

Section: Introductionmentioning

confidence: 93%

Section: Active Speaker Detection (Asd)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

LoCoNet: Long-Short Context Network for Active Speaker Detection

Wang¹,

Cao²,

Bertasius³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

WASD: A Wilder Active Speaker Detection Dataset

Roxo¹,

Costa²,

Inácio³

et al. 2023

Preprint

View full text Add to dashboard Cite

Current Active Speaker Detection (ASD) models achieve great results on AVA-ActiveSpeaker (AVA), using only sound and facial features. Although this approach is applicable in movie setups (AVA), it is not suited for less constrained conditions. To demonstrate this limitation, we propose a Wilder Active Speaker Detection (WASD) dataset, with increased difficulty by targeting the two key components of current ASD: audio and face. Grouped into 5 categories, ranging from optimal conditions to surveillance settings, WASD contains incremental challenges for ASD with tactical impairment of audio and face data. We select state-of-the-art models and assess their performance in two groups of WASD: Easy (cooperative settings) and Hard (audio and/or face are specifically degraded). The results show that: 1) AVA trained models maintain a stateof-the-art performance in WASD Easy group, while underperforming in the Hard one, showing the 2) similarity between AVA and Easy data; and 3) training in WASD does not improve models performance to AVA levels, particularly for audio impairment and surveillance settings. This shows that AVA does not prepare models for wild ASD and current approaches are subpar to deal with such conditions. The proposed dataset also contains body data annotations to provide a new source for ASD, and is available at https://github.com/Tiago-Roxo/WASD.

show abstract

End-to-End Active Speaker Detection

Cited by 2 publications

References 0 publications

LoCoNet: Long-Short Context Network for Active Speaker Detection

LoCoNet: Long-Short Context Network for Active Speaker Detection

WASD: A Wilder Active Speaker Detection Dataset

Contact Info

Product

Resources

About