Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training

Shahmohammadi, Hassan; Lensch, Hendrik P. A.; Baayen, R. Harald

doi:10.18653/v1/2021.conll-1.12

Cited by 14 publications

(9 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the data efficiency gap applies more broadly to language learning. Recent studies evaluating contemporary Transformer-based models have largely re-ported negative results for the effect of multimodality on semantics (Shahmohammadi et al, 2022), commonsense reasoning (Yun et al, 2021), and learning biases (Kuribayashi, 2023). To the best of our knowledge, ours is the first work to perform targeted syntactic evaluation (Marvin and Linzen, 2018;Warstadt et al, 2020a; on multimodal models.…”

Section: Cognitively Oriented Approachesmentioning

confidence: 99%

Acquiring Linguistic Knowledge from Multimodal Input

Amariucai,

Warstadt

2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

In contrast to children, language models (LMs) exhibit considerably inferior data efficiency when acquiring language. In this submission to the BabyLM Challenge (Warstadt et al., 2023), we test the hypothesis that this data efficiency gap is partly caused by a lack of multimodal input and grounding in the learning environment of typical language models. Although previous work looking into this question found that multimodal training can even harm languageonly performance, we speculate that these findings can be attributed to catastrophic forgetting of complex language due to fine-tuning on captions data. To test our hypothesis, we perform an ablation study on FLAVA (Singh et al., 2022), a multimodal vision-and-language model, independently varying the volume of text and vision input to quantify how much text data (if any) can be offset by vision at different data scales. We aim to limit catastrophic forgetting through a multitask pretraining regime that includes unimodal text-only tasks and data sampled from WiT, the relatively diverse Wikipedia-based dataset (Srinivasan et al., 2021). Our results are largely negative: Multimodal pretraining does not harm our models' language performance but does not consistently help either. That said, our conclusions are limited by our having been able to conduct only a small number of runs. While we must leave open the possibility that multimodal input explains some of the gap in data efficiency between LMs and humans, positive evidence for this hypothesis will require better architectures and techniques for multimodal training.

show abstract

Section: Cognitively Oriented Approachesmentioning

confidence: 99%

Acquiring Linguistic Knowledge from Multimodal Input

Amariucai,

Warstadt

2023

Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

show abstract

“…That is, the meanings of words are solely based on other words without links to the outside world. Moreover, they also have practical problems; for example, because of the fundamental assumption of DSM, antonyms belonging to the same topical class (e.g., small and big) typically end up very close together in purely text-based vector spaces (see, e.g., Shahmohammadi et al, 2021, and references cited there). As a consequence, applications for, e.g., sentiment analysis, cannot well distinguish between "it was a good movie" and "it was a bad movie".…”

Section: Introductionmentioning

confidence: 99%

How direct is the link between words and images?

Shahmohammadi,

Heitmeier,

Shafaei-Bajestan

et al. 2023

Self Cite

View full text Add to dashboard Cite

Günther et al. (2022) investigated the relationship between words and images in which they concluded the possibility of a direct link between words and embodied experience. In their study, participants were presented with a target noun and a pair of images, one chosen by their model and another chosen randomly. Participants were asked to select the image that best matched the target noun. Building upon their work, we addressed the following questions. 1. Apart from utilizing visually embodied simulation, what other strategies subjects might have used? How much does this setup rely on visual information? Can it be solved using textual representations? 2. Do current visually-grounded embeddings explain subjects’ selection behavior better than textual embeddings? 3. Does visual grounding improve the representations of both concrete and abstract words? For this aim, we designed novel experiments based on pre-trained word embeddings. Our experiments reveal that subjects’ selection behavior is explained to a large extend on text-based embeddings and word-based similarities. Visually grounded embeddings offered modest advantages over textual embeddings in certain cases. These findings indicate that the experiment by Günther et al. (2022) may not be well suited for tapping into the perceptual experience of participants, and the extent to which it measures visually grounded knowledge is unclear.

show abstract

“…All these methods extract the semantic vectors purely from textual information. Other studies integrate visual information on top of that and create multi-modal embeddings (e.g., Shahmohammadi, Lensch, & Baayen, 2021).…”

Section: Introductionmentioning

confidence: 99%

Semantic properties of English nominal pluralization: Insights from word embeddings

Shafaei-Bajestan¹,

Moradipour-Tari²,

Uhrig³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Semantic differentiation of nominal pluralization is grammaticalized in many languages. For example, plural markers may only be relevant for human nouns. English does not appear to make such distinctions. Using distributional semantics, we show that English nominal pluralization exhibits semantic clusters. For instance, pluralization of fruit words is more similar to one another and less similar to pluralization of other semantic classes. Therefore, reduction of the meaning shift in plural formation to the addition of an abstract plural meaning is too simplistic. A semantically informed method, called CosClassAvg, is introduced that outperforms pluralization methods in distributional semantics which assume plural formation amounts to the addition of a fixed plural vector. In comparison with our approach, a method from compositional distributional semantics, called FRACSS, predicted plural vectors that were more similar to the corpus-extracted plural vectors in terms of direction but not vector length. A modeling study reveals that the observed difference between the two predicted semantic spaces by CosClassAvg and FRACSS carries over to how well a computational model of the listener can understand previously unencountered plural forms. Mappings from word forms, represented with triphone vectors, to predicted semantic vectors are more productive when CosClassAvg-generated semantic vectors are employed as gold standard vectors instead of FRACSS-generated vectors.

show abstract

Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training

Cited by 14 publications

References 46 publications

Acquiring Linguistic Knowledge from Multimodal Input

Acquiring Linguistic Knowledge from Multimodal Input

How direct is the link between words and images?

Semantic properties of English nominal pluralization: Insights from word embeddings

Contact Info

Product

Resources

About