Personality Recognition by Modelling Person-specific Cognitive Processes using Graph Representation

Shao, Zilong; Song, Siyang; Jaiswal, Shashank; Shen, Linlin; Valstar, Michel; Güneş, Hatice

doi:10.1145/3474085.3475460

Cited by 26 publications

(4 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another approach by Shahabinejad et al [40] proposed a CNN architecture to learn and propagate individual deep facial features followed by a spatial attention map, which is then provided as an input to another CNN. For the task of personality computing, Shao et al [41] proposed to learn individual-specifc graph representations for personality traits recognition in a human-human interaction scenario. Individualspecifc CNN architecture is learned from the conversational partner's (speaker) non-verbal cues to predict the target individual's facial reactions (listener).…”

Section: Personalized Models In Human-machine Interactionmentioning

confidence: 99%

Personalized Productive Engagement Recognition in Robot-Mediated Collaborative Learning

Raghuram

Salam

Nasir

et al. 2022

Proceedings of the 2022 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

In this paper, we propose and compare personalized models for Productive Engagement (PE) recognition. PE is defned as the level of engagement that maximizes learning. Previously, in the context of robot-mediated collaborative learning, a framework of productive engagement was developed by utilizing multimodal data of 32 dyads and learning profles, namely, Expressive Explorers (EE), Calm Tinkerers (CT), and Silent Wanderers (SW) were identifed which categorize learners according to their learning gain. Within the same framework, a PE score was constructed in a non-supervised manner for real-time evaluation. Here, we use these profles and the PE score within an AutoML deep learning framework to personalize PE models. We investigate two approaches for this purpose:(1) Single-task Deep Neural Architecture Search (ST-NAS), and (2) Multitask NAS (MT-NAS). In the former approach, personalized models for each learner profle are learned from multimodal features and compared to non-personalized models. In the MT-NAS approach, we investigate whether jointly classifying the learners' profles with the engagement score through multi-task learning would serve as an implicit personalization of PE. Moreover, we compare the predictive power of two types of features: incremental and non-incremental features. Non-incremental features correspond to features computed from the participant's behaviours in fxed time windows. Incremental features are computed by accounting to the behaviour from the beginning of the learning activity till the time window where productive engagement is observed. Our experimental results show that (1) personalized models improve the * Both authors contributed equally to this paper.

show abstract

Section: Personalized Models In Human-machine Interactionmentioning

confidence: 99%

Personalized Productive Engagement Recognition in Robot-Mediated Collaborative Learning

Raghuram

Salam

Nasir

et al. 2022

Proceedings of the 2022 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

show abstract

“…Recent years have witnessed an increasing number of studies targeting human-human dyadic interaction analysis, thanks to the wide application scenarios -such as, among others, surveillance, transportation, health, information security, and intelligent human agent interaction -and the advancement of pattern recognition, cognitive science, and neural networks [16]. Past works [6,19,21,22,25] have investigated the problem of automatically generating an appropriate response or reaction for a given input. Most of those studies focused on the generation of appropriate responses (e.g., using chat-bots [21]) without considering the non-verbal reactions that enrich the message conveyed.…”

Section: Introductionmentioning

confidence: 99%

Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?

Song¹,

Spitale²,

Luo³

et al. 2023

Preprint

View full text Add to dashboard Cite

According to the Stimulus Organism Response (SOR) theory, all human behavioral reactions are stimulated by context, where people will process the received stimulus and produce an appropriate reaction. This implies that in a specific context for a given input stimulus, a person can react differently according to their internal state and other contextual factors. Analogously, in dyadic interactions, humans communicate using verbal and nonverbal cues, where a broad spectrum of listeners' nonverbal reactions might be appropriate for responding to a specific speaker behaviour. There already exists a body of work that investigated the problem of automatically generating an appropriate reaction for a given input. However, none attempted to automatically generate multiple appropriate reactions in the context of dyadic interactions and evaluate the appropriateness of those reactions using objective measures. This paper starts by defining the facial Multiple Appropriate Reaction Generation (fMARG) task for the first time in the literature and proposes a new set of objective evaluation metrics to evaluate the appropriateness of the generated reactions. The paper subsequently introduces a framework to predict, generate, and evaluate multiple appropriate facial reactions.

show abstract

“…Compared with our earlier conference version [30], the extended journal version has following additional contributions and novelties:…”

mentioning

confidence: 99%

Learning Person-Specific Cognition From Facial Reactions for Automatic Personality Recognition

Song

Shao

Jaiswal³

et al. 2023

IEEE Trans. Affective Comput.

View full text Add to dashboard Cite

This paper proposes to recognise the true (self-reported) personality traits from the target subject's cognition simulated from facial reactions. This approach builds on the following two findings in cognitive science: (i) human cognition partially determines expressed behaviour and is directly linked to true personality traits; and (ii) in dyadic interactions, individuals' nonverbal behaviours are influenced by their conversational partner's behaviours. In this context, we hypothesise that during a dyadic interaction, a target subject's facial reactions are driven by two main factors: their internal (person-specific) cognitive process, and the externalised nonverbal behaviours of their conversational partner. Consequently, we propose to represent the target subject's (defined as the listener) person-specific cognition in the form of a person-specific CNN architecture that has unique architectural parameters and depth, which takes audio-visual non-verbal cues displayed by the conversational partner (defined as the speaker) as input, and is able to reproduce the target subject's facial reactions. Each person-specific CNN is explored by the Neural Architecture Search (NAS) and a novel adaptive loss function, which is then represented as a graph representation for recognising the target subject's true personality. Experimental results not only show that the produced graph representations are well associated with target subjects' personality traits in both human-human and human-machine interaction scenarios, and outperform the existing approaches with significant advantages, but also demonstrate that the proposed novel strategies help in learning more reliable personality representations.

show abstract

Personality Recognition by Modelling Person-specific Cognitive Processes using Graph Representation

Cited by 26 publications

References 43 publications

Personalized Productive Engagement Recognition in Robot-Mediated Collaborative Learning

Personalized Productive Engagement Recognition in Robot-Mediated Collaborative Learning

Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?

Learning Person-Specific Cognition From Facial Reactions for Automatic Personality Recognition

Contact Info

Product

Resources

About