UniCon: Unified Context Network for Robust Active Speaker Detection

Abstract

We introduce a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD). Traditional methods for ASD usually operate on each candidate's precropped face track separately and do not sufficiently consider the relationships among the candidates. This potentially limits performance, especially in challenging scenarios with low-resolution faces, multiple candidates, etc. Our solution is a novel, unified framework that focuses on jointly modeling multiple types of contextual information: spatial context to indicate the position and scale of each candidate's face, relational context to capture the visual relationships among the candidates and contrast audio-visual affinities with each other, and temporal context to aggregate longterm information and smooth out local uncertainties. Based on such information, our model optimizes all candidates in a unified process for robust and reliable ASD. A thorough ablation study is performed * Both authors contributed equally to this research.

Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection

“…Roth et al [28] MobileNet [14] 79.2 Zhang et al [38] 3D ResNet-18 [31] + VGG-M [5] 84.0 MAAS-LAN [19] 2D ResNet-18 [13] 85.1 Chung et al [6] VGG-M [5] + 3D Conv 85.5 ASC [2] 2D ResNet-18 [13] 87.1 MAAS-TAN [19] 2D ResNet-18 [13] 88.8 UniCon [39] 2D ResNet-18 [13] 92.0 TalkNet [35] 2D ResNet-18/34 [13] + 3D Conv 92.3 ASDNet [18] 3D ResNeXt-18 [17] + SincDSNet [27] 93.5 SPELL (Ours) 2D ResNet-18-TSM [13,20] 94.2 SPELL+ (Ours) 2D ResNet-50-TSM [13,20] 94.9…”

“…Stage-1 mAP Final mAP ∆mAP MAAS-LAN [19] 79.5 85.1 5.6 ASC [2] 79.5 87.1 7.6 MAAS-TAN [19] 80.2 88.8 8.6 Unicon [39] 84.0 92.0 8.0 ASDNet [18] 88.9…”

“…Another recent work, the ASDNet [18], uses 3D-ResNet101 for encoding visual data and SincNet [27] for audio. The Unified Context Network (UniCon) [39] proposes relational context modules to capture visual (spatial) and audio-visual context based on convolutional layers. Much of these advances are due to the availability of the AVA-ActiveSpeaker dataset [28].…”

“…As visual-only methods [8] are unable to distinguish between verbal and non-verbal lip movements, more recent approaches have focused on joint modeling of the audio-visual information. Audio-visual approaches [2,39,35,19,18] address the task by first encoding visual (primarily facial) and audio features from videos, and then by classifying the fused multimodal features. Such models generally have multi-stage frameworks [2,19,39,18] and show good detection performance.…”

“…Audio-visual approaches [2,39,35,19,18] address the task by first encoding visual (primarily facial) and audio features from videos, and then by classifying the fused multimodal features. Such models generally have multi-stage frameworks [2,19,39,18] and show good detection performance. However, state-of-the-art methods have relied on complex architectures for processing the audio-visual features with high computation and memory overheads.…”