Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding

Yu, Zhou; Yu, Jun; Xiang, Chenchao; Zhao, Zhou; Tian, Qi; Tao, Dacheng

doi:10.24963/ijcai.2018/155

Cited by 116 publications

(68 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They model the conditional probability P (o|r), where r is the referent and o is the appropriate visual object. Instead of modeling P (o|r) directly, others [5], [16], [20], [21], [22], [23], [24] compute P (r|o) by using the CNN-LSTM structure for language generation. The visual region o maximizing P (r|o) is considered to be the target region.…”

Section: Grounding Natural Languagementioning

confidence: 99%

Learning to Compose and Reason with Language Tree Structures for Visual Grounding

Hong

Liu

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

125

View full text Add to dashboard Cite

Grounding natural language in images, such as localizing "the black dog on the left of the tree", is one of the core problems in artificial intelligence, as it needs to comprehend the fine-grained language compositions. However, existing solutions merely rely on the association between the holistic language features and visual features, while neglect the nature of composite reasoning implied in the language. In this paper, we propose a natural language grounding model that can automatically compose a binary tree structure for parsing the language and then perform visual reasoning along the tree in a bottom-up fashion. We call our model RVG-TREE: Recursive Grounding Tree, which is inspired by the intuition that any language expression can be recursively decomposed into two constituent parts, and the grounding confidence score can be recursively accumulated by calculating their grounding scores returned by the two sub-trees. RVG-TREE can be trained end-to-end by using the Straight-Through Gumbel-Softmax estimator that allows the gradients from the continuous score functions passing through the discrete tree construction. Experiments on several benchmarks show that our model achieves the state-of-the-art performance with more explainable reasoning.

show abstract

Section: Grounding Natural Languagementioning

confidence: 99%

Learning to Compose and Reason with Language Tree Structures for Visual Grounding

Hong

Liu

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

125

View full text Add to dashboard Cite

show abstract

“…Each object feature z ∈ Z is then linearly projected into a ranking score s ∈ R and a 4-D bounding box offset b ∈ R 4 , respectively. Similar to [59], we design a multi-task loss function consisting of a ranking loss L rank and a regression loss L reg :…”

Section: Task-specific Headsmentioning

confidence: 99%

Deep Multimodal Neural Architecture Search

Cui

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Designing effective neural networks is fundamentally important in deep multimodal learning. Most existing works focus on a single task and design neural architectures manually, which are highly task-specific and hard to generalize to different tasks. In this paper, we devise a generalized deep multimodal neural architecture search (MMnas) framework for various multimodal learning tasks. Given multimodal input, we first define a set of primitive operations, and then construct a deep encoder-decoder based unified backbone, where each encoder or decoder block corresponds to an operation searched from a predefined operation pool. On top of the unified backbone, we attach task-specific heads to tackle different multimodal learning tasks. By using a gradientbased NAS algorithm, the optimal architectures for different tasks are learned efficiently. Extensive ablation studies, comprehensive analysis, and comparative experimental results show that the obtained MMnasNet significantly outperforms existing state-ofthe-art approaches across three multimodal learning tasks (over five datasets), including visual question answering, image-text matching, and visual grounding. CCS CONCEPTS • Computing methodologies → Multi-task learning; Neural networks.

show abstract

“…In terms of representations of image regions and natural language referring expressions, existing approaches for referring expression comprehension can be generalized into two categories: (1) visual representations un-enriched models, which directly extract deep features from a pretrained CNN as the visual representations of detected image regions (Mao et al, 2016 ; Yu et al, 2016 , 2017 ; Hu et al, 2017 ; Deng et al, 2018 ; Zhang et al, 2018 ; Zhuang et al, 2018 ). (2) visual representations enriched models, which enhance the visual representations by adding external visual information for regions (Liu et al, 2017 ; Yu et al, 2018a , b ). Liu et al ( 2017 ) leveraged external knowledge acquired by an attributes learning model to enrich the information of regions.…”

Section: Related Workmentioning

confidence: 99%

Interactive Natural Language Grounding via Referring Expression Comprehension and Scene Graph Parsing

Lyu

Tang

et al. 2020

Front. Neurorobot.

View full text Add to dashboard Cite

Natural language provides an intuitive and effective interaction interface between human beings and robots. Currently, multiple approaches are presented to address natural language visual grounding for human-robot interaction. However, most of the existing approaches handle the ambiguity of natural language queries and achieve target objects grounding via dialogue systems, which make the interactions cumbersome and time-consuming. In contrast, we address interactive natural language grounding without auxiliary information. Specifically, we first propose a referring expression comprehension network to ground natural referring expressions. The referring expression comprehension network excavates the visual semantics via a visual semantic-aware network, and exploits the rich linguistic contexts in expressions by a language attention network. Furthermore, we combine the referring expression comprehension network with scene graph parsing to achieve unrestricted and complicated natural language grounding. Finally, we validate the performance of the referring expression comprehension network on three public datasets, and we also evaluate the effectiveness of the interactive natural language grounding architecture by conducting extensive natural language query groundings in different household scenarios.

show abstract

Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding

Cited by 116 publications

References 1 publication

Learning to Compose and Reason with Language Tree Structures for Visual Grounding

Learning to Compose and Reason with Language Tree Structures for Visual Grounding

Deep Multimodal Neural Architecture Search

Interactive Natural Language Grounding via Referring Expression Comprehension and Scene Graph Parsing

Contact Info

Product

Resources

About