Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially Aware Language Acquisition

Abstract

This paper presents a self-supervised method for visual detection of the active speaker in a multi-person spoken interaction scenario. Active speaker detection is a fundamental prerequisite for any artificial cognitive system attempting to acquire language in social settings. The proposed method is intended to complement the acoustic detection of the active speaker, thus improving the system robustness in noisy conditions. The method can detect an arbitrary number of possibly overlapping active speakers based exclusively on visual information about their face. Furthermore, the method does not rely on external annotations, thus complying with cognitive development. Instead, the method uses information from the auditory modality to support learning in the visual domain. This paper reports an extensive evaluation of the proposed method using a large multiperson face-to-face interaction dataset. The results show good performance in a speaker dependent setting. However, in a speaker independent setting the proposed method yields a significantly lower performance. We believe that the proposed method represents an essential component of any artificial cognitive system or robotic platform engaging in social interactions.Index Terms-active speaker detection and localization, language acquisition through development, transfer learning, cognitive systems and development

An Active Speaker Detection Method in Videos using Standard Deviations of Color Histogram

“…These researchers were part of the participants at the ActivityNet Challenge 2019 -Task B Active Speaker Detection (AVA) using the AVA ActiveSpeaker dataset. The design in [9] made use of only visual cues to determine active speakers in videos weakly supervised by the audio stream for automatic labelling of the image frames. Once the face image frames were labeled through stochastic optimization, features were extracted using a CNN which were classi ed experimenting with a non-temporal (Perceptron) and a temporal (LSTM model).…”

“…ASD seeks to classify if a given face at a given time in a video is speaking or not [1] [2]. Active speaker determination proves useful in a number of tasks such as human-computer/human-robot interactions [3], where in a eld of view of multiple speakers, a robot needs to know who is talking in order to turn its head or x it's gaze in that direction to visually pay attention and better manage conversations with the different speakers [4], audio-visual diarization (auto annotation of descriptions in video scenes) [5], allowing deaf audience to better appreciate movies [6], video conferencing systems to allow zooming in on the current speaker [7], a necessary step in the auto curation of audio samples from videos where the face image of the subjects are known [8], speaker naming where in addition to detecting the active speaker, the identity is also made known [6], speech enhancement, video re-targeting for meetings [1] and is a basic prerequisite for arti cial cognitive systems in the acquisition of language in social settings [9]. Research in active speaker detection from videos is faced with challenges such as presence of multiple people leading to variability of possible speakers in a video, poor resolution [10], visibility of speaker in video (speakers who are off screen) [11], faces turned at inconvenient in-plane angles to the recording camera, recordings from YouTube are from varying demographics, have different illumination settings and faces are occluded in some cases.…”

“…However, there are no hardand-fast rules in this research area as researchers have tried other unique approaches and have used varying metrics for the evaluation of their algorithms, for example the mean Average Precision (mAP) [1,24], Lc performance that provides evaluation from an energy perspective [25], F-Score [26], ratio of correctly predicted samples to total number of test samples [27] and area under Receiver Operating Characteristic (auROC) curve [1,10]. Some researchers have used only facial cues [3,4,9,28,29], others have used just audio cues for example [5] and others have used a combination of both cues [1,2]. Some researchers in addition to facial cues have used head movements, hand gestures and prosody [4,28], yet others such as [3,[30][31][32][33] in order to determine active speakers, rely on the use of an array of multiple microphones and cameras because such setup provides directional and spatial information respectively, the problem with such methods apart from the extra overhead is that in most real-life scenarios such as YouTube videos they are not applicable.…”

Bio-Inspired Modality Fusion for Active Speaker Detection

“…Such techniques are successful, for instance, as a means to recognize speech related facial motion while also discerning it from other unrelated movements. In [16,19], Stefanov et al used the outputs of a face detector, along with audio-derived voice activity detection (VAD) labels, to train a CNN task specific feature extractor together with a Perceptron classifier. For transfer learning comparison, known CNN architectures were used as pre-trained feature extractors whose outputs were employed in training temporal (LSTM net) and nontemporal (Perceptron) classifiers.…”

Object Detection Recognition and Robot Grasping Based on Machine Learning: A Survey

“…Robot [136,[174][175][176][177][178] Computer vision [135,136,[178][179][180][181] Natural language processing [182,183] Reinforcement…”