Multi-modal social signal analysis for predicting agreement in conversation settings

Ponce-López, Víctor; Escalera, Sérgio; Baró, Xavier

doi:10.1145/2522848.2532594

Cited by 16 publications

(12 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The heterogeneous approach to performance reporting hinders the community from accumulating knowledge regarding the maturity of ML in the field. MMLA researchers are employing different baselines and often reporting the model’s average performance without giving an uncertainty measure [ 13 , 28 , 29 ]. The use of baseline performance only offers a lower bound of performance, which is not sufficient to understand the practical value of the model.…”

Section: Model Evaluation In Machine Learning and Multimodal Learning Analyticsmentioning

confidence: 99%

“…The majority of MMLA researchers [ 20 , 21 , 22 , 26 , 28 , 29 ] have evaluated their models for instance generalizability level while employing various ML models; e.g., SVM, AdaBoost, random forest, neural network and naive Bayes. Among these, random forest is frequently found to be a better model by researchers [ 13 , 17 , 25 ].…”

Section: Efar-mmlamentioning

confidence: 99%

See 1 more Smart Citation

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

Chejara

Prieto

Ruiz‐Calleja

et al. 2021

Sensors

View full text Add to dashboard Cite

Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

show abstract

Section: Model Evaluation In Machine Learning and Multimodal Learning Analyticsmentioning

confidence: 99%

Section: Efar-mmlamentioning

confidence: 99%

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

Chejara

Prieto

Ruiz‐Calleja

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…This paper provides a methodology, which we are using in our challenge on automatic personality trait analysis from video data [26]. The automatic analysis of videos to characterize human behavior has become an area of active research with a wide range of applications [1,2,27,28]. Research advances in computer vision and pattern recognition have lead to methodologies that can successfully recognize consciously executed actions, or intended movements, for instance, gestures, actions, interactions with objects and other people [29].…”

Section: Application Setting: the Design Of A Challengementioning

confidence: 99%

Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits

Chen¹,

Escalera²,

Guyon³

et al. 2016

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

We address the problem of calibration of workers whose task is to label patterns with continuous variables, which arises for instance in labeling images of videos of humans with continuous traits. Worker bias is particularly difficult to evaluate and correct when many workers contribute just a few labels, a situation arising typically when labeling is crowd-sourced. In the scenario of labeling short videos of people facing a camera with personality traits, we evaluate the feasibility of the pairwise ranking method to alleviate bias problems. Workers are exposed to pairs of videos at a time and must order by preference. The variable levels are reconstructed by fitting a Bradley-Terry-Luce model with maximum likelihood. This method may at first sight, seem prohibitively expensive because for N videos, p = N (N − 1)/2 pairs must be potentially processed by workers rather that N videos. However, by performing extensive simulations, we determine an empirical law for the scaling of the number of pairs needed as a function of the number of videos in order to achieve a given accuracy of score reconstruction and show that the pairwise method is affordable. We apply the method to the labeling of a large scale dataset of 10,000 videos used in the ChaLearn Apparent Personality Trait challenge.

show abstract

“…Shan et al [21] use the fusion of facial expressions and body gestures at the feature level and derive an "affect" space by performing Canonical Correlation Analysis. In [18], the authors apply feature extraction techniques to multi-modal audio-RGB-depth data. They compute a set of behavioral indicators that defines communicative cues coming from the fields of psychology and observational methodology.…”

Section: Related Workmentioning

confidence: 99%

Positive/Negative Emotion Detection from RGB-D Upper Body Images

Ballihi

Lablack

Amor

et al. 2015

Face and Facial Expression Recognition From Real World Videos

View full text Add to dashboard Cite

The ability to identify users'mental states represents a valuable asset for improving human-computer interaction. Considering that spontaneous emotions are conveyed mostly through facial expressions and the upper Body movements, we propose to use these modalities together for the purpose of negative/positive emotion classification. A method that allows the recognition of mental states from videos is proposed. Based on a dataset composed with RGB-D movies a set of indictors of positive and negative is extracted from 2D (RGB) information. In addition, a geometric framework to model the depth flows and capture human body dynamics from depth data is proposed. Due to temporal changes in pixel and depth intensity which characterize spontaneous emotions dataset, the depth features are used to define the relation between changes in upper body movements and the affect. We describe a space of depth and texture information to detect the mood of people using upper body postures and their evolution across time. The experimentation has been performed on Cam3D dataset and has showed promising results.

show abstract

Multi-modal social signal analysis for predicting agreement in conversation settings

Cited by 16 publications

References 16 publications

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits

Positive/Negative Emotion Detection from RGB-D Upper Body Images

Contact Info

Product

Resources

About