Learning Abstract Concept Embeddings from Multi-Modal Data: Since You Probably Can't See What I Mean

Abstract

Models that acquire semantic representations from both linguistic and perceptual input are of interest to researchers in NLP because of the obvious parallels with human language learning. Performance advantages of the multi-modal approach over language-only models have been clearly established when models are required to learn concrete noun concepts. However, such concepts are comparatively rare in everyday language. In this work, we present a new means of extending the scope of multi-modal models to more commonly-occurring abstract lexical concepts via an approach that learns multimodal embeddings. Our architecture outperforms previous approaches in combining input from distinct modalities, and propagates perceptual information on concrete concepts to abstract concepts more effectively than alternatives. We discuss the implications of our results both for optimizing the performance of multi-modal models and for theories of abstract conceptual representation.

Bilingual Distributed Word Representations from Document-Aligned Comparable Data

“…While we have already presented two shuffling strategies in this work, one line of future work will investigate different possibilities of "blending in" words from two different vocabularies into pseudo-bilingual documents in a more structured and systematic manner. For instance, one approach to generating pseudo-training sentences for learning from textual and perceptual modalities has been recently introduced (Hill & Korhonen, 2014). However, it is not straightforward how to extend this approach to the generation of pseudo-bilingual training documents.…”

Visually Grounded Meaning Representations

“…For each concept (e.g., bear or eggplant ), we inspected the images in the development set and chose all visual attributes that applied. If an attribute was generally true for the concept, but the images did not provide enough evidence, the attribute was diet (34) eatscarrotsh eats carrots eatsnutsh eats nuts eatsgrassh eats grass shape/size (49) isroundh nevertheless chosen and labeled with <no_evidence>. For example, a plum has a pit, but most images in ImageNet show plums where only the outer part of the fruit is visible.…”

“…Several models ( [15], [19], [22]) present extensions of Latent Dirich-let Allocation (LDA, [42]) where topic distributions are learned from words and other perceptual units treating them both as observed variables. Hill and Korhonen [34] extend the skip-gram network model [33] in a similar fashion, perceptual input is encoded verbally and treated as a word's linguistic context, whereas Lazaridou et al [40] modify skipgram's learning objective so that representations are trained to predict linguistic and visual features. In most cases the visual and textual modalities are decoupled and obtained independently, i.e., from text corpora and feature norms or image databases (but see [19] for an exception).…”

“…In other work ( [39], [40]) representations for the visual modality are obtained directly from image pixels using the feature extraction layers of a deep convolutional neural network (CNN) trained on a large labeled object recognition data set. Finally, some models use human generated image tags as a proxy for visual information ( [20], [34]). …”

“…Some models ( [15], [16], [34]) use feature norms as an approximation of the perceptual environment. Other models focus on the visual modality and exploit image databases, such as ImageNet [35] or ESP [36].…”