Audio-Visual Kinship Verification in the Wild

Abstract

Visual kinship verification entails confirming whether or not two individuals in a given pair of images or videos share a hypothesized kin relation. As a generalized face verification task, visual kinship verification is particularly difficult with low-quality found Internet data. Due to uncontrolled variations in background, pose, facial expression, blur, illumination and occlusion, state-of-the-art methods fail to provide high level of recognition accuracy. As with many other visual recognition tasks, kinship verification may benefit from combining visual and audio signals. However, voice-based kinship verification has received very little prior attention. We hypothesize that the human voice contains kin-related cues that are complementary to visual cues. In this paper we address, for the first time, the use of audio-visual information from face and voice modalities to perform kinship verification. We first propose a new multi-modal kinship dataset, called TALking KINship (TALKIN), that contains several pairs of Internet-quality video sequences. Using TALKIN, we then study the utility of various kinship verification methods including traditional local feature based methods (e.g. LBP, LPQ, etc.) and statistical methods (e.g., GMM-UBM and i-vector), and more recent deep learning approaches (e.g., VGG, LSTM and ResNet 50). Then, early and late fusion methods are evaluated on the TALKIN dataset for the study of kinship verification with both face and voice modalities. Finally, we propose a deep Siamese fusion network with contrastive loss for multi-modal fusion of kinship relations. Extensive experiments on the TALKIN dataset indicate that by combining face and voice modalities, the proposed Siamese network can provide a significantly higher level of accuracy compared to baseline uni-modal and multi-modal fusion techniques. Our experiments show that we can obtain an EER of 40.1% by using the voice modality, an EER of 32.5% by using the face modality, and, by combining face and voice modalities with the deep Siamese fusion network, we can achieve an EER of 29.8%. Experimental results also indicate that audio (vocal) information is complementary (to facial information) and useful for kinship verification.

Audio-Visual Kinship Verification: a New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach

“…2) Facial features: We consider four traditional facial descriptors,BSIF [58], LPQ [57], LBP [56], LBP-TOP [59], as has been done in [20]. We also implement the metric learning method MNRML [3] that combines multiple feature descriptors, LBP, LPQ, and BSIF features, to learn the multi-view data metric.…”

“…The facial video kinship datasets are the ones that only the facial information is available, including UvA-NEMO Smile [32], KFVW [33] and KIVI [34]. The video and audio kinship datasets include TALKIN dataset [20] and FIW MM dataset [35]. The TALKIN dataset is the first audiovisual kinship dataset.…”

“…Therefore, we first adopt the modality-specific networks to transform the face and voice data into the latent feature space. Following the work in [20], the network inputs are the facial image and spectrogram computed from a particular speech. The residual network (ResNet) architecture [48] is adopted for both face and voice backbone network described as below.…”

“…High quality datasets enable rapid progress in the FKV task. In our previous preliminary attempt [20], we have collected the TALKIN dataset. However, the TALKIN has some obvious limitations, i.e., limited number of training samples, limited diversity in terms of environmental conditions, kinship categories, and mono-annotation with binary kinship labels only.…”

“…• We address a new task of audio-visual kinship verification and investigate to exploit the human voice as the complimentary feature to solve the FKV problem. This is an extended version of our conference paper [20]. We improve our previous work from aspects of the proposed dataset and evaluated methods, including benchmarks and the proposed fusion method.…”