MAAS: Multi-modal Assignation for Active Speaker Detection

Alcázar, Juan León; Heilbron, Fabian Caba; Thabet, Ali; Ghanem, Bernard

doi:10.1109/iccv48922.2021.00033

Cited by 28 publications

(27 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Unified Context Network (Unicon) [74] proposes relational context modules to capture visual (spatial) and audiovisual context based on convolutional layers. MAAS-TAN [71] proposes a different multimodal graph approach. Following MAAS-TAN, SPELL [73] presents a model that achieved superior performance by proposing an efficient graph-based framework.…”

Section: Related Workmentioning

confidence: 99%

“…As shown in Table4, to attain a state-of-the-art result, recently, various ASD works that employ graph-based, contextual, and three-stage frameworks have been introduced. MAA-TAN [71] employed graph neural networks approach. SPELL [73] proposed a learning graph-based representation that can significantly improve the active speaker detection performance owing to its explicit spatial and temporal structure.…”

Section: Comparison With State-of-the-artmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Audiovisual Fusion for Active Speaker Detection

Tesema

Song

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Active speaker detection (ASD) refers to detecting the speaking person among visible human instances in a video. Existing methods widely employed a similar audiovisual fusion approach, the concatenation. Although such a fusion approach is often argued to help enhance performance, it must be noted that neither feature modalities play an equal role. It forces the backend network to focus on learning intramodal rather than intermodal features. Another concern is that since the concatenation doubles the fused feature dimension that feeds from the audio and video module, it creates a higher computational overhead for the backend network. To address these problems, this work hypothesizes that instead of leveraging deterministic fusion operation, employing an efficient fusion technique may assist the network in learning efficiently and improve detection accuracy. This work proposes an efficient audiovisual fusion (AVF) with fewer feature dimensions that captures the correlations between facial regions and sound signals, focusing more on the discriminative facial features and associating them with the corresponding audio features. Furthermore, previous ASD works focus only on improving ASD performance by creating a large computational overhead using complex techniques such as adding sophisticated postprocessing, applying smoothing techniques on the classifier to refine the network outputs at multiple stages, or assembling the multiple network outputs. This work proposed a simple yet effective end-to-end ASD using the newly proposed feature fusion approach, the AVF. The proposed framework attained a mAP of 84.384% on the validation set of the most challenging audiovisual speaker detection benchmark, the AVA-ActiveSpeaker. With this, this work outperformed previous works that did not apply the postprocessing tasks and attained competitive detection accuracy compared to other works that employed different postprocessing tasks. The proposed model also learns better on the unsynchronized raw AVA-ActiveSpeaker dataset. The ablation experiments under different image scale settings and noisy signals show the AFV's effectiveness and robustness than the concatenation operation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Comparison With State-of-the-artmentioning

confidence: 99%

Efficient Audiovisual Fusion for Active Speaker Detection

Tesema

Song

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Alcázar et al [1,2] first exploit the temporal contextual and relational contextual information from multiple speakers to handle the active speaker detection task. Köpüklü et al [18] and Min et al [22] follow this idea to design structures that can better model temporal and relational contexts to improve detection performance.…”

Section: Related Workmentioning

confidence: 99%

“…mAP vs. FLOPs, size ∝ parameters. The mAP of different active speaker detection methods [1,2,18,22,36,44] on the benchmark and the FLOPs required to predict one frame containing three candidates. The size of the blobs is proportional to the number of model parameters.…”

Section: Introductionmentioning

confidence: 99%

“…With the release of the first large-scale active speaker detection dataset AVA-ActiveSpeaker [32], researchers have made a series of significant progress in this field [15,36,37,39,46] following the rapid development of deep learning for audio-visual tasks [21]. These studies improve the performance of active speaker detection by inputting face sequences of multiple candidates at the same time [1,2,46], extracting visual features with 3D convolutional neural networks [3,18,47], modeling cross-modal information with complex attention modules [9,43,44], etc, which brings higher memory and computation requirements. Therefore, existing works are difficult to be applied in scenarios requiring real-time processing with limited memory and computational resources, such as automatic video editing and live television.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Light Weight Model for Active Speaker Detection

Liao¹,

Duan²,

Kanghui³

et al. 2023

Preprint

View full text Add to dashboard Cite

Active speaker detection is a challenging task in audiovisual scenario understanding, which aims to detect who is speaking in one or more speakers scenarios. This task has received extensive attention as it is crucial in applications such as speaker diarization, speaker tracking, and automatic video editing. The existing studies try to improve performance by inputting multiple candidate information and designing complex models. Although these methods achieved outstanding performance, their high consumption of memory and computational power make them difficult to be applied in resource-limited scenarios. Therefore, we construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%), while the resource costs are significantly lower than the state-of-the-art method, especially in model parameters (1.0M vs. 22.5M, about 23×) and FLOPs (0.6G vs. 2.6G, about 4×). In addition, our framework also performs well on the Columbia dataset showing good robustness. The code and model weights are available at https: //github.com/Junhua-Liao/Light-ASD.

show abstract

Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection

Min

Roy

Tripathi

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We address the problem of active speaker detection through a new framework, called SPELL, that learns long-range multimodal graphs to encode the inter-modal relationship between audio and visual data. We cast active speaker detection as a node classification task that is aware of longer-term dependencies. We first construct a graph from a video so that each node corresponds to one person. Nodes representing the same identity share edges between them within a defined temporal window. Nodes within the same video frame are also connected to encode inter-person interactions. Through extensive experiments on the Ava-ActiveSpeaker dataset, we demonstrate that learning graph-based representation, owing to its explicit spatial and temporal structure, significantly improves the overall performance. SPELL outperforms several relevant baselines and performs at par with state of the art models while requiring an order of magnitude lower computation cost.

show abstract

MAAS: Multi-modal Assignation for Active Speaker Detection

Cited by 28 publications

References 36 publications

Efficient Audiovisual Fusion for Active Speaker Detection

Efficient Audiovisual Fusion for Active Speaker Detection

A Light Weight Model for Active Speaker Detection

Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection

Contact Info

Product

Resources

About