Integrating Cross-modal Interactions via Latent Representation Shift for Multi-modal Humor Detection

Chen, Chengxin; Zhang, Pengyuan

doi:10.1145/3551876.3554805

Cited by 12 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Participants could utilise the text, audio, and video modality. Several systems were proposed to tackle this task, all of them employing the Transformer architecture [49,50,51].…”

Section: Multimodal Humour Recognitionmentioning

confidence: 99%

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Christ¹,

Amiriparian²,

Kathan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Humour is a substantial element of human affect and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection are solely based on staged data making them inadequate for 'real-world' applications. We address this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among various subjects, highlighting the individuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. Finally, we make our code publicly available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset is available upon request.

show abstract

“…Participants could utilise the text, audio, and video modality. Several systems were proposed to tackle this task, all of them employing the Transformer architecture [49,50,51].…”

Section: Multimodal Humour Recognitionmentioning

confidence: 99%

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Christ¹,

Amiriparian²,

Kathan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The training set comprises recordings from German football press conferences, while the unseen test set exclusively features press conferences conducted in English, thus setting the stage for a cross-cultural, cross-lingual evaluation scenario. For the training partition, we utilize the German Passau Spontaneous Football Coach Humor (P SFCH) (P SFCH) dataset [21], previously featured in the 2022 and 2023 editions of MuSe [19,33,40,65,66,70]. For the unseen test partition, we expanded the P SFCH dataset by incorporating press conference recordings delivered by seven distinct coaches from the English Premier League spanning from September 2016 to September 2020.…”

Section: Introductionmentioning

confidence: 99%

MuSe 2022 Challenge

Amiriparian

Christ

König

et al. 2022

Proceedings of the 30th ACM International Conference on Multimedia

View full text Add to dashboard Cite

The Multimodal Sentiment Analysis Challenge (MuSe) 2024 addresses two contemporary multimodal affect and sentiment analysis problems: In the Social Perception Sub-Challenge (M S P), participants will predict 16 different social attributes of individuals such as assertiveness, dominance, likability, and sincerity based on the provided audio-visual data. The Cross-Cultural Humor Detection Sub-Challenge (M S H) dataset expands upon the Passau Spontaneous Football Coach Humor (P SFCH) dataset, focusing on the detection of spontaneous humor in a cross-lingual and cross-cultural setting. The main objective of MuSe 2024 is to unite a broad audience from various research domains, including multimodal sentiment analysis, audio-visual affective computing, continuous signal processing, and natural language processing. By fostering collaboration and exchange among experts in these fields, the MuSe 2024 endeavors to advance the understanding and application of sentiment analysis and affective computing across multiple modalities. This baseline paper provides details on each subchallenge

show abstract

“…Different from the training data, the unseen test set consists of videos of English football press conferences, thus providing a cross-cultural evaluation setting. The Passau Spontaneous Football Coach Humour (Passau-SFCH) dataset [18] as featured in MuSe 2022 [17,38,63] is provided as the training set. For the test set, we extend Passau-SFCH with recordings of press conferences given by 7 different coaches from the English Premier League between September 2016 and September 2020.…”

mentioning

confidence: 99%

The MuSe 2022 Multimodal Sentiment Analysis Challenge

Christ

Amiriparian

Baird³

et al. 2022

Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge

View full text Add to dashboard Cite

nnnnnnn feature sets. As a competitive baseline system, a Gated Recurrent Unit (GRU)-Recurrent Neural Network (RNN) is employed. On the respective sub-challenges' test datasets, it achieves a mean (across three continuous intensity targets) Pearson's Correlation Coefficient of .4727 for MuSe-Mimic, an Area Under the Curve (AUC) value of .8310 for MuSe-Humour and Concordance Correlation Coefficient (CCC) values of .7482 for arousal and .7827 for valence in the MuSe-Personalisation sub-challenge. CCS CONCEPTS• Computing methodologies → Neural networks; Artificial intelligence; Computer vision; Natural language processing.

show abstract

Integrating Cross-modal Interactions via Latent Representation Shift for Multi-modal Humor Detection

Cited by 12 publications

References 12 publications

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results

MuSe 2022 Challenge

The MuSe 2022 Multimodal Sentiment Analysis Challenge

Contact Info

Product

Resources

About