MultiMediate '23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions

Müller, Philipp; Balazia, Michal; Baur, Tobias; Dietz, Michael; Heimerl, Alexander; Schiller, Dominik; Guermal, Mohammed; Thomas, Dominike; Brémond, François; Alexandersson, Jan; André, Elisabeth; Bulling, Andreas

doi:10.1145/3581783.3613851

Cited by 13 publications

(4 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we present the outcomes of our experiments and compare with the baseline method [13]. Given the presence of imbalanced class samples in this 14-class multi-label classification problem, we apply mean average precision as the evaluation metric.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Madan,

Jain,

Sharma

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Bodily behavioral language is an important social cue, and its automated analysis helps in enhancing the understanding of artificial intelligence systems. Furthermore, behavioral language cues are essential for active engagement in social agent-based user interactions. Despite the progress made in computer vision for tasks like head and body pose estimation, there is still a need to explore the detection of finer behaviors such as gesturing, grooming, or fumbling. This paper proposes a multiview attention fusion method named MAGIC-TBR that combines features extracted from videos and their corresponding Discrete Cosine Transform coefficients via a transformer-based approach. The experiments are conducted on the BBSI dataset and the results demonstrate the effectiveness of the proposed feature fusion with multiview attention. The code is available at: https://github.com/surbhimadan92/MAGIC-TBR CCS CONCEPTS• Computing methodologies → Machine learning; • Humancentered computing → Empirical studies in HCI.

show abstract

Section: Resultsmentioning

confidence: 99%

“…This integration leads to the creation of robust video-level features. We demonstrate through extensive experiments on the benchmark dataset [1] that our MAGIC-TBR approach improves bodily behavior recognition performance compared to the baseline method [13].…”

mentioning

confidence: 99%

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Madan,

Jain,

Sharma

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…We evaluated the proposed model on the backchannel detection dataset in the MultiMediate'23 challenge [15] that was originally introduced in MultiMediate'22 [16]. The experimental results demonstrate the significance of attending to the salient moments and visual body parts, and the effectiveness of attention modules.…”

Section: Introductionmentioning

confidence: 99%

“…The ability to accurately detect backchannel occurrence has profound implications across a wide range of domains, such as artificial mediation [9,12,22]. This paper proposes a novel approach for the MultiMediate backchannel detection challenge [15,16].…”

Section: Introductionmentioning

confidence: 99%

Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks

Wang,

Cheung,

Zhang

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

Automatic detection of backchannel has great potential to enhance artificial mediators, which indicate listeners' attention and agreement in human communication. It is often expressed by subtle nonverbal cues that occur briefly and sparsely. Focusing on identifying and locating these subtle cues (i.e., their occurrence moment and the involved body parts), this paper proposes a novel approach for backchannel detection. In particular, our model utilizes temporaland modality-attention modules to determine and lead the model to pay more attention to both the indicative moment and the accompanying body parts at that specific time. It achieves an accuracy of 68.6% on the testing set in MultiMediate'23 backchannel detection challenge, outperforming the counterparts. Furthermore, we conducted an ablation study to thoroughly understand the contributions of our model. This study underscores the effectiveness of our selection of modality inputs and the importance of the two attention modules in our model. CCS CONCEPTS• Computing methodologies → Artificial intelligence.

show abstract

A Cross-Multi-modal Fusion Approach for Enhanced Engagement Recognition

Dresvyanskiy,

Karpov,

Minker

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

MultiMediate '23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions

Cited by 13 publications

References 59 publications

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings

Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks

A Cross-Multi-modal Fusion Approach for Enhanced Engagement Recognition

Contact Info

Product

Resources

About