Previous studies in multimodal sentiment analysis have used limited datasets, which only contain unified multimodal annotations. However, the unified annotations do not always reflect the independent sentiment of single modalities and limit the model to capture the difference between modalities. In this paper, we introduce a Chinese single-and multimodal sentiment analysis dataset, CH-SIMS, which contains 2,281 refined video segments in the wild with both multimodal and independent unimodal annotations. It allows researchers to study the interaction between modalities or use independent unimodal annotations for unimodal sentiment analysis. Furthermore, we propose a multi-task learning framework based on late fusion as the baseline. Extensive experiments on the CH-SIMS show that our methods achieve state-of-the-art performance and learn more distinctive unimodal representations. The full dataset and codes are available for use at https://github.com/ thuiar/MMSA.

Multimodal sentiment analysis is an emerging research field that aims to enable machines to recognize, interpret, and express emotion. Through the cross-modal interaction, we can get more comprehensive emotional characteristics of the speaker. Bidirectional Encoder Representations from Transformers (BERT) is an efficient pre-trained language representation model. Fine-tuning it has obtained new state-of-the-art results on eleven natural language processing tasks like question answering and natural language inference. However, most previous works fine-tune BERT only base on text data, how to learn a better representation by introducing the multimodal information is still worth exploring. In this paper, we propose the Cross-Modal BERT (CM-BERT), which relies on the interaction of text and audio modality to fine-tune the pre-trained BERT model. As the core unit of the CM-BERT, masked multimodal attention is designed to dynamically adjust the weight of words by combining the information of text and audio modality. We evaluate our method on the public multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results show that it has significantly improved the performance on all the metrics over previous baselines and text-only finetuning of BERT. Besides, we visualize the masked multimodal attention and proves that it can reasonably adjust the weight of words by introducing audio modality information. CCS CONCEPTS • Information systems → Sentiment analysis; Multimedia and multimodal retrieval; • Computing methodologies → Natural language processing;

No abstract

The challenge of assessment in collaborative learning is well known. The major issue is whether the assessment should focus on the individual level or the group level. Traditional assessment approaches only concern about individualized assessment, examination, or coding individual transcripts into speech acts. How to support teachers to monitor and assess collaborative learning processes at the group level has not been systematically addressed. This paper aims to design a new approach for assessing collaborative learning processes and group performance through the lens of knowledge convergence. We use the innovative knowledge map approach to analyze the degree of process and outcome convergence so as to provide insight into the qualities of collaborative learning processes. A total of 94 undergraduate students participated in this study. The empirical results indicate that the qualities of process can be quantified by the number of activated common knowledge and the degree of process convergence. The degree of outcome convergence can be used as an effective indicator for assessing group performance. Implications for instructors to facilitate knowledge convergence in collaborative learning are also discussed. The major contribution of this study is the design of a 167-185 DOI 10.1007/s40692-014-0009-7 novel approach to assess a collaborative learning process and outcome by analyzing the degree of knowledge convergence.