Hierarchical Context-Based Emotion Recognition With Scene Graphs

Wu, Shichao; Zhou, Lei; Hu, Zhengxi; Liu, Jingtai

doi:10.1109/tnnls.2022.3196831

Cited by 10 publications

(6 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Graph representations can also help modeling context, incorporating contextual cues in the interpretation of affect recognition. For instance, in [139], graphs are used to learn the affective relationships between different context elements in a picture, and in [140], a hierarchical method based on scene graphs is presented. Following a similar reasoning, Gao et al [141] propose a novel framework based on a GCN that leverages both spatial and temporal contextual features for video emotion recognition, outperforming SOTA methods for context-aware emotion recognition while allowing a direct visualization between final emotion predictions and salient regions in video frames.…”

Section: Graphsmentioning

confidence: 99%

Toward Explainable Affective Computing: A Review

Cortiñas-Lorenzo,

Lacey

2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Affective computing has an unprecedented potential to change the way humans interact with technology. While the last decades have witnessed vast progress in the field, multimodal affective computing systems are generally black box by design. As affective systems start to be deployed in real-world scenarios, such as education or healthcare, a shift of focus toward improved transparency and interpretability is needed. In this context, how do we explain the output of affective computing models? and how to do so without limiting predictive performance? In this article, we review affective computing work from an explainable AI (XAI) perspective, collecting and synthesizing relevant papers into three major XAI approaches: premodel (applied before training), in-model (applied during training), and postmodel (applied after training). We present and discuss the most fundamental challenges in the field, namely, how to relate explanations back to multimodal and time-dependent data, how to integrate context and inductive biases into explanations using mechanisms such as attention, generative modeling, or graph-based methods, and how to capture intramodal and cross-modal interactions in post hoc explanations. While explainable affective computing is still nascent, existing methods are promising, contributing not only toward improved transparency but, in many cases, surpassing state-of-the-art results. Based on these findings, we explore directions for future research and discuss the importance of data-driven XAI and explanation goals, and explainee needs definition, as well as causability or the extent to which a given method leads to human understanding.

show abstract

Section: Graphsmentioning

confidence: 99%

Toward Explainable Affective Computing: A Review

Cortiñas-Lorenzo,

Lacey

2024

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Annotations are made based on the apparent emotional states of the subjects depicted in the images have discussed the importance of considering the person's scene in the problem of automatic emotion recognition in the wild using EMOTIC database presented a CNN-based three-stream deep hybrid framework that combines the proposed visual feature type with the features extracted from the entire image and the primary human subject. This approach aims to optimize the integration of the proposed feature with the visual cues from the entire image and the main subject (Wu et al, 2022) proposed a hierarchical relation-based emotion recognition method using scene graphs inspired by humans' advanced reasoning patterns. In the scene, entities are labeled, and their relationships are described abstractly.…”

Section: Image-based Machine Learning Classifiers In Measurement Of S...mentioning

confidence: 99%

Bi-Model Engagement Emotion Recognition Based on Facial and Upper-Body Landmarks and Machine Learning Approaches

Alhasson,

Alsaheel,

Alharbi

et al. 2023

International Journal of E-Services and Mobile Applications

View full text Add to dashboard Cite

Customer satisfaction can be measured using facial expression recognition. The current generation of artificial intelligence systems heavily depends on facial features such as eyebrows, eyes, and foreheads. This dependence introduces a limitation as people generally prefer to conceal their genuine emotions. As body gestures are difficult to conceal and can convey a more detailed and accurate emotional state, the authors incorporate upper-body gestures as an additional feature that improves the predicted emotion's accuracy. This work uses an ensemble machine-learning model that integrates support vector machines, random forest classifiers, and logistic regression classifiers. The proposed method detects emotions from facial expressions and upper-body movements and is experimentally evaluated and has been found to be effective, with an accuracy rate of 97% on the EMOTIC dataset and 99% accuracy on MELD dataset.

show abstract

“…Furthermore, if we organize the process of charisma generation into levels of abstraction, methods such as hierarchical RL (Saleh et al, 2020;Rothenpieler and Amiriparian, 2023) can offer a valuable approach for modeling charismatic behaviors at various levels of complexity. It can begin from overarching attributes like empathy and humor (Christ et al, 2022a;Kathan et al, 2022) (highlevel) to specific behaviors such as affective storytelling (Christ et al, 2022b) and body language (Wu et al, 2022) (mid-level), down to fine-grained details like speech patterns (Niebuhr et al, 2016;Amiriparian and Schuller, 2021) and linguistics (Rosenberg and Hirschberg, 2009;Fraser et al, 2022) (low-level). This multilevel approach could enhance the comprehensiveness of charisma generation and capture its multifaceted nature.…”

Section: Learning-based Generation Of Charismamentioning

confidence: 99%

Computational charisma—A brick by brick blueprint for building charismatic artificial intelligence

Schuller,

Amiriparian,

Batliner

et al. 2023

Front. Comput. Sci.

View full text Add to dashboard Cite

Charisma is considered as one's ability to attract and potentially influence others. Clearly, there can be considerable interest from an artificial intelligence's (AI) perspective to provide it with such skill. Beyond, a plethora of use cases opens up for computational measurement of human charisma, such as for tutoring humans in the acquisition of charisma, mediating human-to-human conversation, or identifying charismatic individuals in big social data. While charisma is a subject of research in its own right, a number of models exist that base it on various “pillars,” that is, dimensions, often following the idea that charisma is given if someone could and would help others. Examples of such pillars, therefore, include influence (could help) and affability (would help) in scientific studies, or power (could help), presence, and warmth (both would help) as a popular concept. Modeling high levels in these dimensions, i. e., high influence and high affability, or high power, presence, and warmth for charismatic AI of the future, e. g., for humanoid robots or virtual agents, seems accomplishable. Beyond, also automatic measurement appears quite feasible with the recent advances in the related fields of Affective Computing and Social Signal Processing. Here, we therefore present a brick by brick blueprint for building machines that can appear charismatic, but also analyse the charisma of others. We first approach the topic very broadly and discuss how the foundation of charisma is defined from a psychological perspective. Throughout the manuscript, the building blocks (bricks) then become more specific and provide concrete groundwork for capturing charisma through artificial intelligence (AI). Following the introduction of the concept of charisma, we switch to charisma in spoken language as an exemplary modality that is essential for human-human and human-computer conversations. The computational perspective then deals with the recognition and generation of charismatic behavior by AI. This includes an overview of the state of play in the field and the aforementioned blueprint. We then list exemplary use cases of computational charismatic skills. The building blocks of application domains and ethics conclude the article.

show abstract

Hierarchical Context-Based Emotion Recognition With Scene Graphs

Cited by 10 publications

References 55 publications

Toward Explainable Affective Computing: A Review

Toward Explainable Affective Computing: A Review

Bi-Model Engagement Emotion Recognition Based on Facial and Upper-Body Landmarks and Machine Learning Approaches

Computational charisma—A brick by brick blueprint for building charismatic artificial intelligence

Contact Info

Product

Resources

About