AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Abstract

Audio-visual speaker diarization aims at detecting "who spoken when" using both auditory and visual signals. Existing audio-visual diarization datasets are mainly focused on indoor environments like meeting rooms or news studios, which are quite different from in-the-wild videos in many scenarios such as movies, documentaries, and audience sitcoms. To create a testbed that can effectively compare diarization methods on videos in the wild, we annotate the speaker diarization labels on the AVA movie dataset and create a new benchmark called AVA-AVD. This benchmark is challenging due to the diverse scenes, complicated acoustic conditions, and completely off-screen speakers. Yet, how to deal with off-screen and on-screen speakers together still remains a critical challenge. To overcome it, we propose a novel Audio-Visual Relation Network (AVR-Net) which introduces an effective modality mask to capture discriminative information based on visibility. Experiments have shown that our method not only can outperform state-of-the-art methods but also is more robust as varying the ratio of off-screen speakers. Ablation studies demonstrate the advantages of the proposed AVR-Net and especially the modality mask on diarization. Our data and code will be made publicly available at https://github.com/zcxueric/AVA-AVD.

Audio-visual Speaker Diarization: Improved Voice Activity Detection with CNN based Feature Extraction

“…First, the shot boundaries in the video are detected. 1 The shots, in practice, correspond to sequences of frames that look similar concerning some similarity measures. After that, the frames in each shot are grouped in face tracks and feature vectors are extracted from them.…”

“…Finally, for every cluster, an identity label is assigned for verifying the active speaker in the next module of our pipeline [21]. 1 We relied on PySceneDetect framework for the implementation of shot detection (https://github.com/Breakthrough/PySceneDetect).…”

“…at the University of Thessaly, Volos, Greece. Besides audio-only, modern approaches [1], [2] also exploit visual information by employing audio-visual fusion techniques [3], [4] to synchronize audio and visual information and enhance the performance of diarization systems. For example, in [5], audio and frame-based visual activity features are combined using a weighted likelihood of a Gaussian Mixture Model (GMM).…”