Is Someone Speaking?

Tao, Ruijie; Pan, Zexu; Das, Rohan Kumar; Qian, Xinyuan; Shou, Mike Zheng; Li, Haizhou

doi:10.1145/3474085.3475587

Cited by 91 publications

(65 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is the author's version which has not been fully edited and To address the limitations of SCMIA, we propose to incorporate the AV-activity information, which can complement the speaker's identity information. We use AV-activity-based models for ASD such as TalkNet [25] and Syncnet [11] to gather the concerning AV-activity information.…”

Section: Volume mentioning

confidence: 99%

“…For GSCMIA, we employ two strategies to gather AVactivity ASD predictions: i) TalkNet [25] and ii) Syncnet [11]. Talknet [25] is one of the state-of-the-art methods that observe a face track and the concurrent audio waveform and models the short-term and long-term temporal context to provide face-box-wise active speaker predictions. It has been trained in a fully-supervised manner using the active speaker annotations in each frame.…”

Section: Experiments and Implementation Details A Implementation Detailsmentioning

confidence: 99%

“…In Table 1, we show the performance of SCMIA and TalkNet [25] and observe that TalkNet performs significantly better than SCMIA for the AVA dataset. At the same time, the performance is comparable for both strategies in the case of the TV shows: Friends and TBBT.…”

Section: B Performance Evaluationmentioning

confidence: 99%

“…In these methods, researchers model the AV-activity using sophisticated neural networks consisting of 3D CNNs and RNNs [12], [22]. With further advancements in deep neural networks, researchers have modeled the AV-activity's short-term and long-term temporal dynamics using graph-convolutional neural networks and self-attention neural networks [23]- [25]. Researchers have also proposed to utilize context in the form of visual activity : of the co-occurring speakers to enhance the performance of ASD further [26].…”

Section: Introductionmentioning

confidence: 99%

“…We evaluate the proposed approach on three datasets: the AVA-active speaker dataset [19], Friends, and TBBT (part of the Visual Person Clustering Dataset [21]) consisting of videos from movies and TV shows. We show experiments using two different AV-activity-based systems, TalkNet [25] and Syncnet [11]. The contributions of this work are as follows:…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection

Sharma

Narayanan

2023

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

Active speaker detection in videos addresses associating a source face, visible in the video frames, with the underlying speech in the audio modality. The two primary sources of information to derive such a speech-face relationship are i) visual activity and its interaction with the speech signal and ii) cooccurrences of speakers' identities across modalities in the form of face and speech. The two approaches have their limitations: the audio-visual activity models get confused with other frequently occurring vocal activities, such as laughing and chewing, while the speakers' identity-based methods are limited to videos having enough disambiguating information to establish a speech-face association. Since the two approaches are independent, we investigate their complementary nature in this work. We propose a novel unsupervised framework to guide the speakers' cross-modal identity association with the audio-visual activity for active speaker detection. Through experiments on entertainment media videos from two benchmark datasets--the AVA active speaker (movies) and Visual Person Clustering Dataset (TV shows)--we show that a simple late fusion of the two approaches enhances the active speaker detection performance.

show abstract

Section: Volume mentioning

confidence: 99%

Section: Experiments and Implementation Details A Implementation Detailsmentioning

confidence: 99%

Section: B Performance Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection

Sharma

Narayanan

2023

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

show abstract

Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detection

Min

Roy

Tripathi

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We address the problem of active speaker detection through a new framework, called SPELL, that learns long-range multimodal graphs to encode the inter-modal relationship between audio and visual data. We cast active speaker detection as a node classification task that is aware of longer-term dependencies. We first construct a graph from a video so that each node corresponds to one person. Nodes representing the same identity share edges between them within a defined temporal window. Nodes within the same video frame are also connected to encode inter-person interactions. Through extensive experiments on the Ava-ActiveSpeaker dataset, we demonstrate that learning graph-based representation, owing to its explicit spatial and temporal structure, significantly improves the overall performance. SPELL outperforms several relevant baselines and performs at par with state of the art models while requiring an order of magnitude lower computation cost.

show abstract