Machine Learning Methods for Social Signal Processing

Rudovic, Ognjen; Nicolaou, Mihalis A.; Pavlović, Vladimir

doi:10.1017/9781316676202.018

Cited by 12 publications

(9 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To recognize the signifcance of an expression researchers must note where an expression is displayed, when it is displayed and who the presenter is [65]. Later, researchers included the signifcance of why and how a cue is expressed [49]. Research in this feld has been successful in capturing postures [55], gestures [11], vocal behaviour [15] and inferring emotions from facial expression and eye movements [68].…”

Section: Related Workmentioning

confidence: 99%

Communication Skills Training Intervention Based on Automated Recognition of Nonverbal Signals

Pereira

Hone

2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

There have been promising studies that show a potential of providing social signal feedback to improve communication skills. However, these studies have primarily focused on unimodal methods of feedback. In addition to this, studies do not assess whether skills are maintained after a given time. With a sample size of 22 this paper investigates whether multimodal social signal feedback is an efective method of improving communication in the context of media interviews. A pre-post experimental evaluation of media skills training intervention is presented which compares standard feedback with augmented feedback based on automated recognition of multimodal social signals. Results revealed signifcantly diferent training efects between the two conditions. However, the initial experiment study failed to show signifcant diferences in human judgement of performance. A 6-month follow-up study revealed human judgement ratings were higher for the experiment group. This study suggests that augmented selective multimodal social signal feedback is an efective method for communication skills training. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Empirical studies in HCI.

show abstract

Section: Related Workmentioning

confidence: 99%

Communication Skills Training Intervention Based on Automated Recognition of Nonverbal Signals

Pereira

Hone

2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…Recently, automated facial expression analysis has attracted increasing interest within the computer vision and machine learning communities [2], [5]. As facial expressions can be decomposed in a physical and affective component, its automatic analysis can also be separated on either of the two components.…”

Section: Related Workmentioning

confidence: 99%

“…At its foundation are a set of fundamental computer vision and machine learning problems that have largely driven the great progress that those fields have made during the past few decades. From a computer vision perspective, facial expression analysis from image sequences is a very challenging task due to the complex deformation of the face, the loss of 3D information during the image formation process [4] and the presence of nuisance factors such as person-specific morphology, view-point variations and unknown lighting conditions [5]. From a machine learning perspective, one of the most challenging problems is the ground truth problem [6].…”

Section: Introductionmentioning

confidence: 99%

Leveraging the Bayesian Filtering Paradigm for Vision-Based Facial Affective State Estimation

Oveneke

González

Enescu

et al. 2018

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

Estimating a person's affective state from facial information is an essential capability for social interaction. Automatizing such a capability has therefore increasingly driven multidisciplinary research for the past decades. At the heart of this issue are very challenging signal processing and artificial intelligence problems driven by the inherent complexity of human affect. We therefore propose a principled framework for designing automated systems capable of continuously estimating the human affective state from an incoming stream of images. First, we model human affect as a dynamical system and define the affective state in terms of valence, arousal and their higher-order derivatives. We then pose the affective state estimation problem as a Bayesian filtering problem and provide a solution based on Kalman filtering (KF) for probabilistic reasoning over time, combined with multiple instance sparse Gaussian processes (MI-SGP) for inferring affect-related measurements from image sequences. We quantitatively and qualitatively evaluate our proposed framework on the AVEC 2012 and AVEC 2014 benchmark datasets and obtain state-of-the-art results using the baseline features as input to our MI-SGP-KF model. We therefore believe that leveraging the Bayesian filtering paradigm can pave the way for further enhancing the design of automated systems for affective state estimation.

show abstract

“…At first, we automatically detect listener behaviors such as backchannels, laughing, head nodding, and eye gaze from signals of multimodal sensors. Recent machine learning techniques have been applied to this task and achieved sufficient accuracy [21]. According to the observations of the behaviors, the level of engagement is estimated.…”

Section: Introductionmentioning

confidence: 99%

Engagement recognition by a latent character model based on multimodal listener behaviors in spoken dialogue

Inoue

Lala

Takanashi

et al. 2018

SIP

View full text Add to dashboard Cite

Engagement represents how much a user is interested in and willing to continue the current dialogue. Engagement recognition will provide an important clue for dialogue systems to generate adaptive behaviors for the user. This paper addresses engagement recognition based on multimodal listener behaviors of backchannels, laughing, head nodding, and eye gaze. In the annotation of engagement, the ground-truth data often differs from one annotator to another due to the subjectivity of the perception of engagement. To deal with this, we assume that each annotator has a latent character that affects his/her perception of engagement. We propose a hierarchical Bayesian model that estimates both engagement and the character of each annotator as latent variables. Furthermore, we integrate the engagement recognition model with automatic detection of the listener behaviors to realize online engagement recognition. Experimental results show that the proposed model improves recognition accuracy compared with other methods which do not consider the character such as majority voting. We also achieve online engagement recognition without degrading accuracy.

show abstract

Machine Learning Methods for Social Signal Processing

Cited by 12 publications

References 83 publications

Communication Skills Training Intervention Based on Automated Recognition of Nonverbal Signals

Communication Skills Training Intervention Based on Automated Recognition of Nonverbal Signals

Leveraging the Bayesian Filtering Paradigm for Vision-Based Facial Affective State Estimation

Engagement recognition by a latent character model based on multimodal listener behaviors in spoken dialogue

Contact Info

Product

Resources

About