What goes into a word: generating image descriptions with top-down spatial knowledge

Ghanimifard, Mehdi; Dobnik, Simon

doi:10.18653/v1/w19-8668

Cited by 16 publications

(10 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, relations such as “the leg is next to the ear” or “the banana is in front of the nose” will not be learned by CNNs because of the small size of the convolutional kernel while objects and their features (“leg”, “ear”, “banana”, “nose”) will be detected ( Kelleher and Dobnik, 2017 ; Ghanimifard and Dobnik, 2018 ). Thus, one type of knowledge that is encoded by attention weights is the knowledge of long-distance visual dependencies between objects (see, for example, the study by Ghanimifard and Dobnik (2019) ).…”

Section: Methodsmentioning

confidence: 99%

What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

Ilinykh

Dobnik

2021

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

Neural networks have proven to be very successful in automatically capturing the composition of language and different structures across a range of multi-modal tasks. Thus, an important question to investigate is how neural networks learn and organise such structures. Numerous studies have examined the knowledge captured by language models (LSTMs, transformers) and vision architectures (CNNs, vision transformers) for respective uni-modal tasks. However, very few have explored what structures are acquired by multi-modal transformers where linguistic and visual features are combined. It is critical to understand the representations learned by each modality, their respective interplay, and the task’s effect on these representations in large-scale architectures. In this paper, we take a multi-modal transformer trained for image captioning and examine the structure of the self-attention patterns extracted from the visual stream. Our results indicate that the information about different relations between objects in the visual stream is hierarchical and varies from local to a global object-level understanding of the image. In particular, while visual representations in the first layers encode the knowledge of relations between semantically similar object detections, often constituting neighbouring objects, deeper layers expand their attention across more distant objects and learn global relations between them. We also show that globally attended objects in deeper layers can be linked with entities described in image descriptions, indicating a critical finding - the indirect effect of language on visual representations. In addition, we highlight how object-based input representations affect the structure of learned visual knowledge and guide the model towards more accurate image descriptions. A parallel question that we investigate is whether the insights from cognitive science echo the structure of representations that the current neural architecture learns. The proposed analysis of the inner workings of multi-modal transformers can be used to better understand and improve on such problems as pre-training of large-scale multi-modal architectures, multi-modal information fusion and probing of attention weights. In general, we contribute to the explainable multi-modal natural language processing and currently shallow understanding of how the input representations and the structure of the multi-modal transformer affect visual representations.

show abstract

Section: Methodsmentioning

confidence: 99%

What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

Ilinykh

Dobnik

2021

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Regier (1996) designed the neurons to learn the meaning of spatial prepositions. Ghanimifard and Dobnik (2019) explored the effects of spatial knowledge in a generative neural language model for the image description. We mainly work on incorporating the spatial semantics in navigation neural agent.…”

Section: Related Workmentioning

confidence: 99%

Towards Navigation by Reasoning over Spatial Configurations

Zhang¹,

Guo²,

Kordjamshidi³

2021

Proceedings of Second International Combined Workshop on Spatial Language Understanding and Grounded Communication for Robotics

View full text Add to dashboard Cite

We deal with the navigation problem where the agent follows natural language instructions while observing the environment. Focusing on language understanding, we show the importance of spatial semantics in grounding navigation instructions into visual perceptions. We propose a neural agent that uses the elements of spatial configurations and investigate their influence on the navigation agent's reasoning ability. Moreover, we model the sequential execution order and align visual objects with spatial configurations in the instruction. Our neural agent improves strong baselines on the seen environments and shows competitive performance on the unseen environments. Additionally, the experimental results demonstrate that explicit modeling of spatial semantic elements in the instructions can improve the grounding and spatial reasoning of the model.

show abstract

“…Their model does not decode language to 2D spatial arrangements while reasoning about their position. Finally, Ghanimifard and Dobnik (2019) generate spatial image descriptions to investigate what kind of spatial bottom-up knowledge, benefits the top-down methods the most.…”

Section: Related Workmentioning

confidence: 99%

Decoding Language Spatial Relations to 2D Spatial Arrangements

Radevski¹,

Collell²,

Moens³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

We address the problem of multimodal spatial understanding by decoding a set of languageexpressed spatial relations to a set of 2D spatial arrangements in a multi-object and multirelationship setting. We frame the task as arranging a scene of clip-arts given a textual description. We propose a simple and effective model architecture SPATIAL-REASONING BERT (SR-BERT), trained to decode text to 2D spatial arrangements in a non-autoregressive manner. SR-BERT can decode both explicit and implicit language to 2D spatial arrangements, generalizes to out-of-sample data to a reasonable extent and can generate complete abstract scenes if paired with a clip-arts predictor. Finally, we qualitatively evaluate our method with a user study, validating that our generated spatial arrangements align with human expectation.

show abstract

What goes into a word: generating image descriptions with top-down spatial knowledge

Cited by 16 publications

References 34 publications

What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations

Towards Navigation by Reasoning over Spatial Configurations

Decoding Language Spatial Relations to 2D Spatial Arrangements

Contact Info

Product

Resources

About