Speech/non-speech detection in meetings from automatically extracted low resolution visual features

Hung, Hayley; Ba, Silèye

doi:10.1109/icassp.2010.5494913

Cited by 20 publications

(17 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hung and Ba [10] applied visual activity (the amount of movement) and focus of visual attention as features to determine who is the current speaker on real meeting room corpus data. Action units (AU) were used as input features to Hidden Markov Models (HMM) in Stefanov et al [11].…”

Section: Introductionmentioning

confidence: 99%

Vision-based Active Speaker Detection in Multiparty Interaction

Stefanov¹,

Beskow²,

Salvi³

2017

GLU 2017 International Workshop on Grounding Language Understanding

View full text Add to dashboard Cite

This paper presents a supervised learning method for automatic visual detection of the active speaker in multiparty interactions. The presented detectors are built using a multimodal multiparty interaction dataset previously recorded with the purpose to explore patterns in the focus of visual attention of humans. Three different conditions are included: two humans involved in taskbased interaction with a robot; the same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The paper also presents an evaluation of the active speaker detection method in a speaker dependent experiment showing that the method achieves good accuracy rates in a fairly unconstrained scenario using only image data as input. The main goal of the presented method is to provide real-time detection of the active speaker within a broader framework implemented on a robot and used to generate natural focus of visual attention behavior during multiparty human-robot interactions.

show abstract

Section: Introductionmentioning

confidence: 99%

Vision-based Active Speaker Detection in Multiparty Interaction

Stefanov¹,

Beskow²,

Salvi³

2017

GLU 2017 International Workshop on Grounding Language Understanding

View full text Add to dashboard Cite

show abstract

“…Other approaches for speaker detection include a general pattern recognition framework used by Besson and Kunt [30] applied to detection of the speaker in audio-visual sequences. Visual activity (the amount of movement) and focus of visual attention were used as inputs by Hung and Ba [31] to determine the current speaker on real meetings. Stefanov et al [32] used action units as inputs to Hidden Markov Models to determine the active speaker in multi-party interactions and Vajaria et al [33] demonstrated that information for body movements can improve the detection performance.…”

Section: B Active Speaker Detectionmentioning

confidence: 99%

Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially Aware Language Acquisition

Stefanov

Beskow

Salvi

2020

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

This paper presents a self-supervised method for visual detection of the active speaker in a multi-person spoken interaction scenario. Active speaker detection is a fundamental prerequisite for any artificial cognitive system attempting to acquire language in social settings. The proposed method is intended to complement the acoustic detection of the active speaker, thus improving the system robustness in noisy conditions. The method can detect an arbitrary number of possibly overlapping active speakers based exclusively on visual information about their face. Furthermore, the method does not rely on external annotations, thus complying with cognitive development. Instead, the method uses information from the auditory modality to support learning in the visual domain. This paper reports an extensive evaluation of the proposed method using a large multiperson face-to-face interaction dataset. The results show good performance in a speaker dependent setting. However, in a speaker independent setting the proposed method yields a significantly lower performance. We believe that the proposed method represents an essential component of any artificial cognitive system or robotic platform engaging in social interactions.Index Terms-active speaker detection and localization, language acquisition through development, transfer learning, cognitive systems and development

show abstract

“…Actually, cognitive scientists showed that speech and gestures are so tightly intertwined that every important investigation of language has taken gestures into account [51]. While the multimodal diarization is common in the literature (based on the joint modeling of speech, facial and bodily cues), the unimodal diarization exploiting visual information is rare [52], unrelated to surveillance. We think that this is a direction worth to be investigated, because this allows to capture turn taking patterns indicating ongoing conversations and thus genuine social interactions (PROBLEM 3).…”

Section: Gesture and Posturementioning

confidence: 99%

Human behavior analysis in video surveillance: A Social Signal Processing perspective

et al. 2013

View full text Add to dashboard Cite

The analysis of human activities is one of the most intriguing and important open issues for the automated video surveillance community. Since few years ago, it has been handled following a mere Computer Vision and Pattern Recognition perspective, where an activity corresponded to a temporal sequence of explicit actions (run, stop, sit, walk, etc.). Even under this simplistic assumption, the issue is hard, due to the strong diversity of the people appearance, the number of individuals considered (we may monitor single individuals, groups, crowd), the variability of the environmental conditions (indoor/outdoor, different weather conditions), and the kinds of sensors employed. More recently, the automated surveillance of human activities has been faced considering a new perspective, that brings in notions and principles from the social, affective, and psychological literature, and that is called Social Signal Processing (SSP). SSP employs primarily nonverbal cues, most of them are outside of conscious awareness, like face expressions and gazing, body posture and gestures, vocal characteristics, relative distances in the space and the like. This paper is the first review analyzing this new trend, proposing a structured snapshot of the state of the art and envisaging novel challenges in the surveillance domain where the cross-pollination of Computer Science technologies and Sociology theories may offer valid investigation strategies.

show abstract

Speech/non-speech detection in meetings from automatically extracted low resolution visual features

Cited by 20 publications

References 20 publications

Vision-based Active Speaker Detection in Multiparty Interaction

Vision-based Active Speaker Detection in Multiparty Interaction

Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially Aware Language Acquisition

Human behavior analysis in video surveillance: A Social Signal Processing perspective

Contact Info

Product

Resources

About