Learning Visual Voice Activity Detection with an Automatically Annotated Dataset

Abstract

Visual voice activity detection (V-VAD) uses visual features to predict whether a person is speaking or not. V-VAD is useful whenever audio VAD (A-VAD) is inefficient either because the acoustic signal is difficult to analyze or because it is simply missing. We propose two deep architectures for V-VAD, one based on facial landmarks and one based on optical flow. Moreover, available datasets, used for learning and for testing V-VAD, lack content variability. We introduce a novel methodology to automatically create and annotate very large datasets inthe-wild -WildVVAD -based on combining A-VAD with face detection and tracking. A thorough empirical evaluation shows the advantage of training the proposed deep V-VAD models with this dataset. 1

The VVAD-LRS3 Dataset for Visual Voice Activity Detection

“…There seems to be not many datasets for the VVAD task. The only competitive state of the art dataset for VVAD that we found was the WildVVAD [7]. WildVVAD is not only 3 times smaller than the VVAD-LRS3 it is also more prone false positive and false negative because of the loose assumption that detected voice activity and a single face in the video equals a speaking sample and every detected face in a video sequence without voice activity is a not speaking sample.…”

“…WildVVAD is not only 3 times smaller than the VVAD-LRS3 it is also more prone false positive and false negative because of the loose assumption that detected voice activity and a single face in the video equals a speaking sample and every detected face in a video sequence without voice activity is a not speaking sample. Very high 1-to-1 WildVVAD [7] 13,000 High 1-to-1 LRS3 [22] >100,000 High 1-to-0 CUAVE [20] ∼7,000 Low 1-to-0…”