Simple and Effective Multimodal Learning Based on Pre-Trained Transformer Models

Abstract

Transformer-based models have been garnering attention in various fields. Beginning with their success in natural language processing, transformer-based models have succeeded in several other fields, such as image and automatic speech recognition. In addition to them being trained on unimodal information, many transformer-based models have been proposed for multimodal information. In the field of multimodal learning, a common problem encountered is that of insufficient multimodal training data. This study attempted to address this problem by proposing a simple albeit effective method using 1) unimodal pre-trained transformer models as encoders for each modal input, and 2) another set of transformer layers to fuse their output representations. Further, the proposed method was evaluated through a series of experiments on two common benchmarks: CMU-MOSI and MM-IMDb. Consequently, the proposed model obtained state-of-the-art performances in both benchmarks, and its robustness in reducing the amount of training data was demonstrated.

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…Of the 5 papers in the visual domain, classification [376], detection [377], prediction [378], identification [379] and extraction [380] have 1. The entertainment domain has 6 relevant articles, where 2 were in segmentation [381], [382], 1 was in prediction [383], 2 were in detection [384], [385] and 1 was in recognition [386].…”