Multimodal Active Speaker Detection and Virtual Cinematography for Video Conferencing

Cutler, Ross; Mehran, Ramin; Johnson, Samuel; Cha, Zhang; Kirk, Adam G.; Whyte, Oliver; Kowdle, Adarsh

doi:10.1109/icassp40776.2020.9053171

Cited by 8 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has a long history in computer vision [13]. ASD is used in various applications such as automatic video annotation [64], video conferencing [1], human-robot interactions, speech recognition, speaker diarization or re-framing video [11], speech transcription, speech enhancement [13], tracking storylines and characters in narrative content [13], [15] and facilitate the mining of training data for modeling speaker identification [16], [17]. In video conferencing, ASD allows the far-end participants to see who is currently speaking, which is especially useful when the conference room is large or the remote video is rendered on a small display due to small screen size, small render size, or limited bandwidth [1], [13].…”

Section: Introductionmentioning

confidence: 99%

Efficient Audiovisual Fusion for Active Speaker Detection

Tesema

Song

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Active speaker detection (ASD) refers to detecting the speaking person among visible human instances in a video. Existing methods widely employed a similar audiovisual fusion approach, the concatenation. Although such a fusion approach is often argued to help enhance performance, it must be noted that neither feature modalities play an equal role. It forces the backend network to focus on learning intramodal rather than intermodal features. Another concern is that since the concatenation doubles the fused feature dimension that feeds from the audio and video module, it creates a higher computational overhead for the backend network. To address these problems, this work hypothesizes that instead of leveraging deterministic fusion operation, employing an efficient fusion technique may assist the network in learning efficiently and improve detection accuracy. This work proposes an efficient audiovisual fusion (AVF) with fewer feature dimensions that captures the correlations between facial regions and sound signals, focusing more on the discriminative facial features and associating them with the corresponding audio features. Furthermore, previous ASD works focus only on improving ASD performance by creating a large computational overhead using complex techniques such as adding sophisticated postprocessing, applying smoothing techniques on the classifier to refine the network outputs at multiple stages, or assembling the multiple network outputs. This work proposed a simple yet effective end-to-end ASD using the newly proposed feature fusion approach, the AVF. The proposed framework attained a mAP of 84.384% on the validation set of the most challenging audiovisual speaker detection benchmark, the AVA-ActiveSpeaker. With this, this work outperformed previous works that did not apply the postprocessing tasks and attained competitive detection accuracy compared to other works that employed different postprocessing tasks. The proposed model also learns better on the unsynchronized raw AVA-ActiveSpeaker dataset. The ablation experiments under different image scale settings and noisy signals show the AFV's effectiveness and robustness than the concatenation operation.

show abstract

Section: Introductionmentioning

confidence: 99%

Efficient Audiovisual Fusion for Active Speaker Detection

Tesema

Song

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Active Speaker Detection (ASD) refers to the task of identifying when each visible person is speaking in a video, typically through careful joint analysis of face motion and voices. It has a wide range of modern practical applications, such as video re-targeting [14], What marks an active speaker out from others? Admittedly, face motion and its synchrony with the audio are the most obvious clues; however, as shown in the figure above, the underlying signals can be highly ambiguous, especially in hard scenarios with poorly lit, low-resolution faces and noisy acoustics, etc.…”

Section: Introductionmentioning

confidence: 99%

UniCon: Unified Context Network for Robust Active Speaker Detection

Zhang

Liang

Yang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

We introduce a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD). Traditional methods for ASD usually operate on each candidate's precropped face track separately and do not sufficiently consider the relationships among the candidates. This potentially limits performance, especially in challenging scenarios with low-resolution faces, multiple candidates, etc. Our solution is a novel, unified framework that focuses on jointly modeling multiple types of contextual information: spatial context to indicate the position and scale of each candidate's face, relational context to capture the visual relationships among the candidates and contrast audio-visual affinities with each other, and temporal context to aggregate longterm information and smooth out local uncertainties. Based on such information, our model optimizes all candidates in a unified process for robust and reliable ASD. A thorough ablation study is performed * Both authors contributed equally to this research.

show abstract

A Study on Domain Adaptation for Audio-Visual Speech Enhancement

Wang,

Chen,

et al. 2024

Communications in Computer and Information Science

View full text Add to dashboard Cite

Multimodal Active Speaker Detection and Virtual Cinematography for Video Conferencing

Cited by 8 publications

References 12 publications

Efficient Audiovisual Fusion for Active Speaker Detection

Efficient Audiovisual Fusion for Active Speaker Detection

UniCon: Unified Context Network for Robust Active Speaker Detection

A Study on Domain Adaptation for Audio-Visual Speech Enhancement

Contact Info

Product

Resources

About