Multimodal Distributional Semantics

Abstract

Distributional semantic models derive computational representations of word meaning from the patterns of co-occurrence of words in text. Such models have been a success story of computational linguistics, being able to provide reliable estimates of semantic relatedness for the many semantic tasks requiring them. However, distributional models extract meaning information exclusively from text, which is an extremely impoverished basis compared to the rich perceptual sources that ground human semantic knowledge. We address the lack of perceptual grounding of distributional models by exploiting computer vision techniques that automatically identify discrete “visual words” in images, so that the distributional representation of a word can be extended to also encompass its co-occurrence with the visual words of images it is associated with. We propose a flexible architecture to integrate text- and image-based distributional information, and we show in a set of empirical tests that our integrated model is superior to the purely text-based approach, and it provides somewhat complementary semantic information with respect to the latter.

Visually Grounded Meaning Representations

“…The second model is a deep learning-based variant of kCCA (DCCA, [82]) which computes representations by passing the two views through a deep network which is fine-tuned to maximize the total correlation of the output layers. The third model emulates Bruni et al's [37] integration mechanism. Specifically, we concatenated the textual and visual vectors and projected them onto a lower dimensional latent space using singular value decomposition, a mathematical technique for reducing the dimensionality of semantic spaces [83].…”

“…Other models focus on the visual modality and exploit image databases, such as ImageNet [35] or ESP [36]. A few approaches ( [19], [37]) use visual words which they derive by clustering SIFT descriptors [38] extracted from images, or combine both feature norms and visual words [22]. Drawing inspiration from the successful application of attribute classifiers in object recognition, Silberer et al [21] show that automatically predicted visual attributes from images can act as substitutes for feature norms without any critical information loss.…”

“…Other approaches infer bimodal representations over latent variables responsible for the co-occurrence of words over featural dimensions. Bruni et al [37] concatenate two independently constructed textual and visual spaces and subsequently project them onto a lower-dimensional space using Singular Value Decomposition (SVD). Several models ( [15], [19], [22]) present extensions of Latent Dirich-let Allocation (LDA, [42]) where topic distributions are learned from words and other perceptual units treating them both as observed variables.…”

“…We further report results with Bruni et al's [37] bimodal distributional model using their publicly available system [84]. Their textual modality is represented by a 30K-dimensional co-occurrence matrix 14 extracted from the ukWaC corpus (2 billion tokens) 15 and WaCkypedia.…”

Addressing Low-Resource Scenarios with Character-aware Embeddings

“…CAT assigns each word the concatenation of its CL, WL, and FT embeddings, i.e., performs late fusion (Bruni et al, 2014). As the individual models produce vectors of different average magnitudes, we rescale the embeddings produced by individual models prior to this concatenation, such that, after rescaling, each constituent model has the same average vector magnitude, when averaged over all words present in the training data.…”