MSCTD: A Multimodal Sentiment Chat Translation Dataset

Abstract

Multimodal machine translation and textual chat translation have received considerable attention in recent years. Although the conversation in its natural form is usually multimodal, there still lacks work on multimodal machine translation in conversations. In this work, we introduce a new task named Multimodal Chat Translation (MCT), aiming to generate more accurate translations with the help of the associated dialogue history and visual context. To this end, we firstly construct a Multimodal Sentiment Chat Translation Dataset (MSCTD) containing 142,871 English-Chinese utterance pairs in 14,762 bilingual dialogues and 30,370 English-German utterance pairs in 3,079 bilingual dialogues. Each utterance pair, corresponding to the visual context that reflects the current conversational scene, is annotated with a sentiment label. Then, we benchmark the task by establishing multiple baseline systems that incorporate multimodal and sentiment features for MCT. Preliminary experiments on four language directions (English↔Chinese and English↔German) verify the potential of contextual and multimodal information fusion and the positive impact of sentiment on the MCT task. Additionally, as a by-product of the MSCTD, it also provides two new benchmarks on multimodal dialogue sentiment analysis. Our work can facilitate research on both multimodal chat translation and multimodal dialogue sentiment analysis. 1

SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)

“…Recent work in sentiment analysis focused on sub-tasks that tackle new challenges, including aspect-based (Chen et al, 2022), multimodal (Liang et al, 2022), explainable (Cambria et al, 2022), and multilingual sentiment analysis . On the other hand, standard sentiment analysis sub-tasks such as polarity classification (positive, negative, neutral) are widely considered saturated and almost solved (Poria et al, 2020), with an accuracy of 97.5% in certain domains (Raffel et al, 2020;Jiang et al, 2020).…”

AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages

“…Recent work in sentiment analysis focused on subtasks that tackle new challenges, including aspect-based (Chen et al, 2022), multimodal (Liang et al, 2022), explainable (neuro-symbolic) (Cambria et al, 2022), and multilingual sentiment analysis . On the other hand, standard sentiment analysis subtasks such as polarity classification (positive, negative, neutral) are widely considered saturated and solved (Poria et al, 2020), with an accuracy of 97.5% in certain domains (Jiang et al, 2020;Raffel et al, 2020).…”

Scheduled Multi-task Learning for Neural Chat Translation

“…The chat translator bilaterally converts the language of bilingual conversational text, e.g. from Chinese to English and vice versa (Wang et al, 2016a;Farajian et al, 2020;Liang et al, 2021aLiang et al, , 2022. Generally, since the bilingual dialogue corpus is scarce, researchers (Bao et al, 2020;Liang et al, 2021a,e) resort to making use of the large-scale general-domain data through the pre-training-then-fine-tuning paradigm as done in many context-aware neural machine translation models (Miculicich et al, 2018;Tiedemann and Scherrer, 2017;Maruf et al, 2019;Voita et al, 2018Voita et al, , 2019aTu et al, 2018;Ma et al, 2020, etc), having made significant progress.…”