Strong and Simple Baselines for Multimodal Utterance Embeddings

Abstract

Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various attention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a conditional factorization of the utterance into unimodal factors. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the optimal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer representations, our second baseline extends the first by factorizing into unimodal, bimodal, and trimodal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning.

Towards Debiasing Sentence Representations

“…Our text corpora originate from the following five sources: 1) WikiText-2 (Merity et al, 2017a), a dataset of formally written Wikipedia articles (we only use the first 10% of WikiText-2 which we found to be sufficient to capture formally written text), 2) Stanford Sentiment Treebank (Socher et al, 2013), a collection of 10000 polarized written movie reviews, 3) Reddit data collected from discussion forums related to politics, electronics, and relationships, 4) MELD (Poria et al, 2019), a large-scale multimodal multi-party emotional dialog dataset collected from the TV-series Friends, and 5) POM (Park et al, 2014), a dataset of spoken review videos collected across 1,000 individuals spanning multiple topics. These datasets have been the subject of recent research in language understanding (Merity et al, 2017b;Liu et al, 2019; and multimodal human language (Liang et al, 2018(Liang et al, , 2019. Table 2 summarizes these datasets.…”

Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents

“…There exist various exciting recent work on improved multimodal fusion techniques Liang et al, 2019a;Pham et al, 2019;Baltrušaitis et al, 2019). In addition to the simplified feature and modality concatenations, we plan to explore some of these promising tensor-based multimodal fusion networks (Liu et al, 2018;Liang et al, 2019b;Tsai et al, 2019) for more robust intent classification on AMIE dataset as future work.…”

Second Grand-Challenge and Workshop on Multimodal Language (Challenge-HML)

“…There exist various exciting recent work on improved multimodal fusion techniques Liang et al, 2019a;Pham et al, 2019;Baltrušaitis et al, 2019). In addition to the simplified feature and modality concatenations, we plan to explore some of these promising tensor-based multimodal fusion networks Liang et al, 2019b;…”