MAttNet: Modular Attention Network for Referring Expression Comprehension

Yu, Licheng; Lin, Zhe; Shen, Xiaohui; Yang, Jimei; Lu, Xin; Bansal, Mohit; Berg, Tamara L.

doi:10.1109/cvpr.2018.00142

Cited by 733 publications

(780 citation statements)

References 33 publications

Supporting

Mentioning

754

Contrasting

Order By: Relevance

“…The other is the joint vision-language embedding framework to model P(q, r ). During training, the supervision is object proposal and referring expression pairs (r i , q i ) [3,20,25,34,38,46]. The relationship between the target entity and context entity is often used to assist grounding the target in supervised REG methods [15,19,23,30,32,49].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding

Liu

Wang

et al. 2019

Proceedings of the 27th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Weakly supervised referring expression grounding (REG) aims at localizing the referential entity in an image according to linguistic query, where the mapping between the image region (proposal) and the query is unknown in the training stage. In referring expressions, people usually describe a target entity in terms of its relationship with other contextual entities as well as visual attributes. However, previous weakly supervised REG methods rarely pay attention to the relationship between the entities. In this paper, we propose a knowledge-guided pairwise reconstruction network (KPRN), which models the relationship between the target entity (subject) and contextual entity (object) as well as grounds these two entities. Specifically, we first design a knowledge extraction module to guide the proposal selection of subject and object. The prior knowledge is obtained in a specific form of semantic similarities between each proposal and the subject/object. Second, guided by such knowledge, we design the subject and object attention module to construct the subject-object proposal pairs. The subject attention excludes the unrelated proposals from the candidate proposals. The object attention selects the most suitable proposal as the contextual proposal. Third, we introduce a pairwise attention and an adaptive weighting scheme to learn the correspondence between these proposal pairs and the query. Finally, a pairwise reconstruction module is used to measure the grounding for weakly supervised learning. Extensive experiments on four large-scale datasets show our method outperforms existing state-of-the-art methods by a large margin 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

“…Thus we also add an attribute classification branch in our model. The attribute label is extracted through an external language parser [17] according to [46]. Subject feature r i s of proposal is used for attribute classification.…”

Section: Attribute Classification Lossmentioning

confidence: 99%

Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding

Liu

Wang

et al. 2019

Proceedings of the 27th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…[7] decomposes the expression into subjectrelationship-object triplets and aligns the textual representations with image regions using localization module or relationship module; however, referring expressions have much richer forms than this fixed subject-relationship-object template. MattNet [29] decomposes the expressions into three phrases which are corresponding to the subject, location and relationship modules respectively; however, it cannot process multi-step reasoning. The other work [32] enables reasoning as a step-wise attention process following the stepwise representation of the expression; however, it treats the expression as the sequence of words, which ignores the linguistic structure of the expression.…”

Section: Interpretable Reasoningmentioning

confidence: 99%

“…Referring expressions are complex, and include rich dependencies and nested linguistic structures, which further guide the visual reasoning process. In theory, natural language parsers can parse grammatical relations among the words in an expression, but existing language parsers are not practical for referring expression comprehension due to highly unrestricted language [29]. Each complex expression is defined by its constituent expressions and the rules used to combine them.…”

Section: Language-guided Visual Reasoning Processmentioning

confidence: 99%

Dynamic Graph Attention for Referring Expression Comprehension

Yang

Li²,

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

199

View full text Add to dashboard Cite

Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to the grounding of complex referring expressions. In this paper, we explore the problem of referring expression comprehension from the perspective of language-driven visual reasoning, and propose a dynamic graph attention network to perform multi-step reasoning by modeling both the relationships among the objects in the image and the linguistic structure of the expression. In particular, we construct a graph for the image with the nodes and edges corresponding to the objects and their relationships respectively, propose a differential analyzer to predict a language-guided visual reasoning process, and perform stepwise reasoning on top of the graph to update the compound object representation at every node. Experimental results demonstrate that the proposed method can not only significantly surpass all existing state-of-the-art algorithms across three common benchmark datasets, but also generate interpretable visual evidences for stepwisely locating the objects referred to in complex language descriptions.

show abstract

“…The predicted region corresponds to the caption that is ranked most similar to the referring expression. Other works based on RPN (Hu et al, 2017) or Faster-RCNN (Yu et al, 2018) have integrated attention mechanisms to decompose the language expressions into multiple sub-parts but use tailored modules for specific sub-tasks making them less fit for our object referral task. Karpathy et al (2014) interpret the inner product between region proposals and sentence fragments as a similarity score, allowing to match them in a bidirectional manner.…”

Section: Related Workmentioning

confidence: 99%

Talk2Car: Taking Control of Your Self-Driving Car

Deruyttere¹,

Vandenhende²,

Grujicic³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

A long-term goal of artificial intelligence is to have an agent execute commands communicated through natural language. In many cases the commands are grounded in a visual environment shared by the human who gives the command and the agent. Execution of the command then requires mapping the command into the physical visual space, after which the appropriate action can be taken. In this paper we consider the former. Or more specifically, we consider the problem in an autonomous driving setting, where a passenger requests an action that can be associated with an object found in a street scene. Our work presents the Talk2Car dataset, which is the first object referral dataset that contains commands written in natural language for self-driving cars. We provide a detailed comparison with related datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref and CLEVR-Ref. Additionally, we include a performance analysis using strong state-ofthe-art models. The results show that the proposed object referral task is a challenging one for which the models show promising results but still require additional research in natural language processing, computer vision and the intersection of these fields. The dataset can be found on our website: http:// macchina-ai.eu/

show abstract

MAttNet: Modular Attention Network for Referring Expression Comprehension

Cited by 733 publications

References 33 publications

Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding

Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding

Dynamic Graph Attention for Referring Expression Comprehension

Talk2Car: Taking Control of Your Self-Driving Car

Contact Info

Product

Resources

About