Automatic reaction emotion estimation in a human–human dyadic setting using Deep Neural Networks

Sham, Abdallah Hussein; Tikka, Pia; Lamas, David; Anbarjafari, Gholamreza

doi:10.1007/s11760-022-02257-5

Cited by 4 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Regarding how sensitive humans are to noticing discrepancies in human facial expressions even in milliseconds, any delay in facial data synchronization may induce distortions to the dataset. The work presented in this paper can be seen in line with the recent studies on dyadic human–robot interaction, for instance [ 30 , 31 , 32 , 33 ].…”

Section: Introductionsupporting

confidence: 82%

“…Based on the previous study [ 33 ], we next applied the Facial Expression Analysis (FEA) model that had worked well for videos over a huge number of frames (more precisely, over 2000 frames). We used this model to compare the above two models.…”

Section: Resultsmentioning

confidence: 99%

“…We used this model to compare the above two models. The authors from [ 33 ] did not use discrete basic emotions, but instead selected the valence scale from the continuous arousal–valence scale. One of the reasons why it was found useful to apply only the valence scale can be understood from Figure 11 .…”

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Sham

Khan

Lamas

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

Emotion recognition is a significant issue in many sectors that use human emotion reactions as communication for marketing, technological equipment, or human–robot interaction. The realistic facial behavior of social robots and artificial agents is still a challenge, limiting their emotional credibility in dyadic face-to-face situations with humans. One obstacle is the lack of appropriate training data on how humans typically interact in such settings. This article focused on collecting the facial behavior of 60 participants to create a new type of dyadic emotion reaction database. For this purpose, we propose a methodology that automatically captures the facial expressions of participants via webcam while they are engaged with other people (facial videos) in emotionally primed contexts. The data were then analyzed using three different Facial Expression Analysis (FEA) tools: iMotions, the Mini-Xception model, and the Py-Feat FEA toolkit. Although the emotion reactions were reported as genuine, the comparative analysis between the aforementioned models could not agree with a single emotion reaction prediction. Based on this result, a more-robust and -effective model for emotion reaction prediction is needed. The relevance of this work for human–computer interaction studies lies in its novel approach to developing adaptive behaviors for synthetic human-like beings (virtual or robotic), allowing them to simulate human facial interaction behavior in contextually varying dyadic situations with humans. This article should be useful for researchers using human emotion analysis while deciding on a suitable methodology to collect facial expression reactions in a dyadic setting.

show abstract

Section: Introductionsupporting

confidence: 82%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Sham

Khan

Lamas

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Lower ISC values for pairs that for the SS group could be due to communicative turn taking (see, e.g., turn taking in question reply setting, Bögels, 2020). In this case, one's response to the stimuli (e.g., a spontaneous smile or a glance toward the partner) is met with a unique reaction from the other person, in turn provoking potentially further nonverbal interchange based on subtle facial expressions, i.e., dyadic reaction emotion (Sham et al, 2022).…”

Section: Dyadic Behaviors Detected In Emotional Soundsmentioning

confidence: 99%

Unraveling dyadic psycho-physiology of social presence between strangers during an audio drama – a signal-analysis approach

Kauttonen,

Paekivi,

Kauramäki

et al. 2023

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

A mere co-presence of an unfamiliar person may modulate an individual’s attentive engagement with specific events or situations to a significant degree. To understand better how such social presence affects experiences, we recorded a set of parallel multimodal facial and psychophysiological data with subjects (N = 36) who listened to dramatic audio scenes alone or when facing an unfamiliar person. Both a selection of 6 s affective sound clips (IADS-2) followed by a 27 min soundtrack extracted from a Finnish episode film depicted familiar and often intense social situations familiar from the everyday world. Considering the systemic complexity of both the chosen naturalistic stimuli and expected variations in the experimental social situation, we applied a novel combination of signal analysis methods using inter-subject correlation (ISC) analysis, Representational Similarity Analysis (RSA) and Recurrence Quantification Analysis (RQA) followed by gradient boosting classification. We report our findings concerning three facial signals, gaze, eyebrow and smile that can be linked to socially motivated facial movements. We found that ISC values of pairs, whether calculated on true pairs or any two individuals who had a partner, were lower than the group with single individuals. Thus, audio stimuli induced more unique responses in those subjects who were listening to it in the presence of another person, while individual listeners tended to yield a more uniform response as it was driven by dramatized audio stimulus alone. Furthermore, our classifiers models trained using recurrence properties of gaze, eyebrows and smile signals demonstrated distinctive differences in the recurrence dynamics of signals from paired subjects and revealed the impact of individual differences on the latter. We showed that the presence of an unfamiliar co-listener that modifies social dynamics of dyadic listening tasks can be detected reliably from visible facial modalities. By applying our analysis framework to a broader range of psycho-physiological data, together with annotations of the content, and subjective reports of participants, we expected more detailed dyadic dependencies to be revealed. Our work contributes towards modeling and predicting human social behaviors to specific types of audio-visually mediated, virtual, and live social situations.

show abstract

Comparative efficacy of histogram-based local descriptors and CNNs in the MRI-based multidimensional feature space for the differential diagnosis of Alzheimer’s disease: a computational neuroimaging approach

Avots,

Jafari,

Ozcinar

et al. 2024

SIViP

View full text Add to dashboard Cite

Automatic reaction emotion estimation in a human–human dyadic setting using Deep Neural Networks

Cited by 4 publications

References 22 publications

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings

Unraveling dyadic psycho-physiology of social presence between strangers during an audio drama – a signal-analysis approach

Comparative efficacy of histogram-based local descriptors and CNNs in the MRI-based multidimensional feature space for the differential diagnosis of Alzheimer’s disease: a computational neuroimaging approach

Contact Info

Product

Resources

About