Supplementary Material: AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection

Xi, Zhonghua; Pantofaru, Caroline; Chaudhuri, Sourish; Klejch, Ondřej; Marvin, Radhika; Gallagher, Andrew; Kaver, Liat; Ramaswamy, Sharadh; Stopczynski, Arkadiusz; Schmid, Cordelia

doi:10.1109/iccvw.2019.00460

Cited by 29 publications

(53 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Action classification datasets include Kinetics, a video dataset for human action classification (Kay et al, 2017), ActivityNet, a video dataset for action classification and temporal localization (Fabian Caba Heilbron and Niebles, 2015), and AVA, a dataset of spatio-temporally localized atomic visual actions (AVA) . Multi-modal AI datasets include AVA-ActiveSpeaker, an audio-visual dataset for speaker detection (Roth et al, 2019), VGG lip reading dataset, an audio-visual dataset for speech recognition and separation , Mosi, a multimodal corpus of sentiment intensity (Zadeh et al, 2017(Zadeh et al, , 2016, and OpenFace, a multi-modal face recognition (Baltrušaitis et al, 2016). The two major advantages of EgoCom are egocentricity and the inclusion of multiple participant's synchronized audio and video, which as we show, simplifies multi-speaker applications.…”

Section: Related Workmentioning

confidence: 99%

EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset

Northcutt

Zha

Lovegrove

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Multi-modal datasets in artificial intelligence (AI) often capture a third-person perspective, but our embodied human intelligence evolved with sensory input from the egocentric, first-person perspective. Towards embodied AI, we introduce the Egocentric Communications (EgoCom) dataset to advance the state-of-the-art in conversational AI, natural language, audio speech analysis, computer vision, and machine learning. EgoCom is a first-of-its-kind natural conversations dataset containing multi-modal human communication data captured simultaneously from the participants' egocentric perspectives. EgoCom includes 38.5 hours of synchronized embodied stereo audio, egocentric video with 240,000 ground-truth, time-stamped word-level transcriptions and speaker labels from 34 diverse speakers. We study baseline performance on two novel applications that benefit from embodied data: (1) predicting turn-taking in conversations and (2) multi-speaker transcription. For (1), we investigate Bayesian baselines to predict turn-taking within 5% of human performance. For (2), we use simultaneous egocentric capture to combine Google speech-to-text outputs, improving global transcription by 79% relative to a single perspective. Both applications exploit EgoCom's synchronous multi-perspective data to augment performance of embodied AI tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset

Northcutt

Zha

Lovegrove

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…An early work in multimodal ASD used TDNN [7], and there has been significant recent work using DNN-based ASD (e.g., [8], [9], [10], [11], [12], [13], [14]). There is now a large dataset created for this task [15] with an ASD competition [10]. However, we are not aware of any ASD (DNN or otherwise) that does low-latency accurate PTZ without large 2D SLL arrays.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Active Speaker Detection and Virtual Cinematography for Video Conferencing

Cutler

Mehran

Johnson

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Active speaker detection (ASD) and virtual cinematography (VC) can significantly improve the remote user experience of a video conference by automatically panning, tilting and zooming of a video conferencing camera: users subjectively rate an expert video cinematographer's video significantly higher than unedited video. We describe a new automated ASD and VC that performs within 0.3 MOS of an expert cinematographer based on subjective ratings with a 1-5 scale. This system uses a 4K wide-FOV camera, a depth camera, and a microphone array; it extracts features from each modality and trains an ASD using an AdaBoost machine learning system that is very efficient and runs in real-time. A VC is similarly trained using machine learning to optimize the subjective quality of the overall experience. To avoid distracting the room participants and reduce switching latency the system has no moving parts -the VC works by cropping and zooming the 4K wide-FOV video stream. The system was tuned and evaluated using extensive crowdsourcing techniques and evaluated on a dataset with N=100 meetings, each 2-5 minutes in length.

show abstract

“…• Television: To create a simulated environment similar to background television sounds, we first extract the audio from four publicly available video datasets: AVA-ActiveSpeaker dataset [26], advertisement dataset [27], and two compilation videos for TV shows "Friends" and "How I met your mother" available in YouTube. For every training utterance, a random segment from one of these four datasets is picked and added with the original signal with 13-20dB SNR.…”

Section: Data Augmentationmentioning

confidence: 99%

Multi-Task Discriminative Training of Hybrid DNN-TVM Model for Speaker Verification with Noisy and Far-Field Speech

et al. 2019

View full text Add to dashboard Cite

The paper aims to address the task of speaker verification with single-channel, noisy and far-field speech by learning an embedding or feature representation that is invariant to different acoustic environments. We approach from two different directions. First, we adopt a newly proposed discriminative model that hybridizes Deep Neural Network (DNN) and Total Variability Model (TVM) with the goal of integrating their strengths. DNN helps learning a unique variable length representation of the feature sequence while TVM accumulates them into a fixed dimensional vector. Second, we propose a multitask training scheme with cross entropy and triplet losses in order to obtain good classification performance as well as distinctive speaker embeddings. The multi-task training is applied on both the DNN-TVM model and state-of-the-art x-vector system. The results on the development and evaluation sets of the VOiCES challenge reveal that the proposed multi-task training helps improving models that are solely based on cross entropy, and it works better with DNN-TVM architecture than x-vector for the current task. Moreover, the multi-task models tend to show complementary relationship with cross entropy models, and thus improved performance is observed after fusion.

show abstract

Supplementary Material: AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection

Cited by 29 publications

References 36 publications

EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset

EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset

Multimodal Active Speaker Detection and Virtual Cinematography for Video Conferencing

Multi-Task Discriminative Training of Hybrid DNN-TVM Model for Speaker Verification with Noisy and Far-Field Speech

Contact Info

Product

Resources

About