Asd-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers

Datta, Gourav; Etchart, Tyler; Yadav, Vivek; Hedau, Varsha; Natarajan, Premkumar; Chang, Shih-Fu

doi:10.1109/icassp43922.2022.9746991

Cited by 9 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, Tao et al [36] achieve superior performance by using cross-attention and self-attention modules to aggregate audio and visual features. Then, based on this work [36], Wuerkaixi et al [43] and Datta et al [9] improve the performance by introducing positional encoding and improving the attention module. In order to better exploit the potential of the attention module, Xiong et al [44] introduce a multi-modal layer normalization to alleviate the distribution misalignment of audio-visual features.…”

Section: Related Workmentioning

confidence: 95%

“…Mel-frequency cepstral coefficients (MFCCs) is one of the most widely used methods in audio recognition, aiming to improve the accuracy of speech activity detection [26]. Therefore, like most existing active speaker detection methods [9,36,37,43,44,46], we extract a 2-dimensional feature map composed of 13-dimensional MFCCs and temporal information from the original audio signal as the input of the audio feature encoder. However, we do not follow the general idea of previous studies using 2D convolutional neural networks to extract audio features.…”

Section: Audio Feature Encodermentioning

confidence: 99%

“…For each method, we copy the results from its original paper or calculate from the open-source code. Some studies [3,9,43,46] are not yet open source, so we only estimate the parameters and FLOPs of their audio-visual encoder. The E2E indicates end-to-end.…”

Section: Loss Functionmentioning

confidence: 99%

“…It is worth noting that the test set is provided for the ActivityNet challenge and is not available. Therefore, we perform performance evaluation on the validation set as in previous works [9,37,43,44,46].…”

Section: Datasetmentioning

confidence: 99%

“…With the release of the first large-scale active speaker detection dataset AVA-ActiveSpeaker [32], researchers have made a series of significant progress in this field [15,36,37,39,46] following the rapid development of deep learning for audio-visual tasks [21]. These studies improve the performance of active speaker detection by inputting face sequences of multiple candidates at the same time [1,2,46], extracting visual features with 3D convolutional neural networks [3,18,47], modeling cross-modal information with complex attention modules [9,43,44], etc, which brings higher memory and computation requirements. Therefore, existing works are difficult to be applied in scenarios requiring real-time processing with limited memory and computational resources, such as automatic video editing and live television.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Light Weight Model for Active Speaker Detection

Liao¹,

Duan²,

Kanghui³

et al. 2023

Preprint

View full text Add to dashboard Cite

Active speaker detection is a challenging task in audiovisual scenario understanding, which aims to detect who is speaking in one or more speakers scenarios. This task has received extensive attention as it is crucial in applications such as speaker diarization, speaker tracking, and automatic video editing. The existing studies try to improve performance by inputting multiple candidate information and designing complex models. Although these methods achieved outstanding performance, their high consumption of memory and computational power make them difficult to be applied in resource-limited scenarios. Therefore, we construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%), while the resource costs are significantly lower than the state-of-the-art method, especially in model parameters (1.0M vs. 22.5M, about 23×) and FLOPs (0.6G vs. 2.6G, about 4×). In addition, our framework also performs well on the Columbia dataset showing good robustness. The code and model weights are available at https: //github.com/Junhua-Liao/Light-ASD.

show abstract

Section: Related Workmentioning

confidence: 95%

Section: Audio Feature Encodermentioning

confidence: 99%

Section: Loss Functionmentioning

confidence: 99%

Section: Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Light Weight Model for Active Speaker Detection

Liao¹,

Duan²,

Kanghui³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

FabuLight-ASD: unveiling speech activity via body language

Carneiro,

Wermter

2024

Neural Comput & Applic

View full text Add to dashboard Cite

Active speaker detection (ASD) in multimodal environments is crucial for various applications, from video conferencing to human-robot interaction. This paper introduces FabuLight-ASD, an advanced ASD model that integrates facial, audio, and body pose information to enhance detection accuracy and robustness. Our model builds upon the existing Light-ASD framework by incorporating human pose data, represented through skeleton graphs, which minimises computational overhead. Using the Wilder Active Speaker Detection (WASD) dataset, renowned for reliable face and body bounding box annotations, we demonstrate FabuLight-ASD’s effectiveness in real-world scenarios. Achieving an overall mean average precision (mAP) of 94.3%, FabuLight-ASD outperforms Light-ASD, which has an overall mAP of 93.7% across various challenging scenarios. The incorporation of body pose information shows a particularly advantageous impact, with notable improvements in mAP observed in scenarios with speech impairment, face occlusion, and human voice background noise. Furthermore, efficiency analysis indicates only a modest increase in parameter count (27.3%) and multiply-accumulate operations (up to 2.4%), underscoring the model’s efficiency and feasibility. These findings validate the efficacy of FabuLight-ASD in enhancing ASD performance through the integration of body pose data. FabuLight-ASD’s code and model weights are available at https://github.com/knowledgetechnologyuhh/FabuLight-ASD.

show abstract

AS-Net: active speaker detection using deep audio-visual attention

Radman,

Laaksonen

2024

Multimed Tools Appl

View full text Add to dashboard Cite

Active Speaker Detection (ASD) aims at identifying the active speaker among multiple speakers in a video scene. Previous ASD models often seek audio and visual features from long video clips with a complex 3D Convolutional Neural Network (CNN) architecture. However, models based on 3D CNNs can generate discriminative spatial-temporal features, but this comes at the expense of computational complexity, and they frequently face challenges in detecting active speakers in short video clips. This work proposes the Active Speaker Network (AS-Net) model, a simple yet effective ASD method tailored for detecting active speakers in relatively short video clips without relying on 3D CNNs. Instead, it incorporates the Temporal Shift Module (TSM) into 2D CNNs, facilitating the extraction of dense temporal visual features without the need for additional computations. Moreover, self-attention and cross-attention schemes are introduced to enhance long-term temporal audio-visual synchronization, thereby improving ASD performance. Experimental results demonstrate that AS-Net outperforms state-of-the-art 2D CNN-based methods on the AVA-ActiveSpeaker dataset and remains competitive with the methods utilizing more complex architectures.

show abstract

Asd-Transformer: Efficient Active Speaker Detection Using Self And Multimodal Transformers

Cited by 9 publications

References 9 publications

A Light Weight Model for Active Speaker Detection

A Light Weight Model for Active Speaker Detection

FabuLight-ASD: unveiling speech activity via body language

AS-Net: active speaker detection using deep audio-visual attention

Contact Info

Product

Resources

About