Relationship-Embedded Representation Learning for Grounding Referring Expressions

Yang, Sibei; Li, Guanbin; Yu, Yizhou

doi:10.1109/tpami.2020.2973983

Cited by 50 publications

(14 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proposed residual graph attention network allows us to reason the underlying relationship for a more complex expression as shown in Figure 2(b). With extensive experiments, the proposed approach achieves better performance than other state-of-the-art graph network-based approaches [4,13,3] and demonstrates its effectiveness.…”

Section: Introductionmentioning

confidence: 92%

Residual Graph Attention Network and Expression-Respect Data Augmentation Aided Visual Grounding

Wang

Chen

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Visual grounding aims to localize a target object in an image based on a given text description. Due to the innate complexity of language, it is still a challenging problem to perform reasoning of complex expressions and to infer the underlying relationship between the expression and the object in an image. To address these issues, we propose a residual graph attention network for visual grounding. The proposed approach first builds an expression-guided relation graph and then performs multi-step reasoning followed by matching the target object. It allows performing better visual grounding with complex expressions by using deeper layers than other graph network approaches. Moreover, to increase the diversity of training data, we perform an expression-respect data augmentation based on copy-paste operations to pairs of source and target images. The proposed approach achieves better performance with extensive experiments than other state-of-the-art graph network-based approaches and demonstrates its effectiveness.

show abstract

Section: Introductionmentioning

confidence: 92%

Residual Graph Attention Network and Expression-Respect Data Augmentation Aided Visual Grounding

Wang

Chen

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…MattNet [12] introduces the modular design and improves the grounding accuracy by better modeling the subject, location, and relation-related language description. Recent studies further improve the two-stage methods by better modeling the object relationships [10], [13], [14], [15], [33], enforcing correspondence learning [34], or making use of phrase co-occurrences [35], [36], [37]. One-stage Methods.…”

Section: Visual Groundingmentioning

confidence: 99%

TransVG: End-to-End Visual Grounding with Transformers

Deng

Yang

Chen

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

232

117

View full text Add to dashboard Cite

In this work, we explore neat yet effective Transformer-based frameworks for visual grounding. The previous methods generally address the core problem of visual grounding, i.e., multi-modal fusion and reasoning, with manually-designed mechanisms. Such heuristic designs are not only complicated but also make models easily overfit specific data distributions. To avoid this, we first propose TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We empirically show that complicated fusion modules can be replaced by a simple stack of Transformer encoder layers with higher performance. However, the core fusion Transformer in TransVG is stand-alone against uni-modal encoders, and thus should be trained from scratch on limited visual grounding data, which makes it hard to be optimized and leads to sub-optimal performance. To this end, we further introduce TransVG++ to make two-fold improvements. For one thing, we upgrade our framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding. For another, we devise Language Conditioned Vision Transformer that removes external fusion modules and reuses the uni-modal ViT for vision-language fusion at the intermediate layers. We conduct extensive experiments on five prevalent datasets, and report a series of state-of-the-art records.

show abstract

“…Unlike single sentences used in most image-language tasks [36,44,32,37], we adopt free-form language descriptions composed of several sentences in our work, where additional challenges such as long-distance phrase correlations emerge. Specifically, we first build a language scene graph G l from the free-form 3D scene description L to capture the rich structure and relationships between the phrases.…”

Section: Language Scene Graph Modulementioning

confidence: 99%

Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud

Feng

Li³

et al. 2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

3D object grounding aims to locate the most relevant target object in a raw point cloud scene based on a freeform language description. Understanding complex and diverse descriptions, and lifting them directly to a point cloud is a new and challenging topic due to the irregular and sparse nature of point clouds. There are three main challenges in 3D object grounding: to find the main focus in the complex and diverse description; to understand the point cloud scene; and to locate the target object. In this paper, we address all three challenges. Firstly, we propose a language scene graph module to capture the rich structure and long-distance phrase correlations. Secondly, we introduce a multi-level 3D proposal relation graph module to extract the object-object and object-scene co-occurrence relationships, and strengthen the visual features of the initial proposals. Lastly, we develop a description guided 3D visual graph module to encode global contexts of phrases and proposals by a nodes matching strategy. Extensive experiments on challenging benchmark datasets (ScanRefer [3] and Nr3D [42]) show that our algorithm outperforms existing state-of-the-art. Our code is available at https://github.com/PNXD/FFL-3DOG.

show abstract

Relationship-Embedded Representation Learning for Grounding Referring Expressions

Cited by 50 publications

References 54 publications

Residual Graph Attention Network and Expression-Respect Data Augmentation Aided Visual Grounding

Residual Graph Attention Network and Expression-Respect Data Augmentation Aided Visual Grounding

TransVG: End-to-End Visual Grounding with Transformers

Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud

Contact Info

Product

Resources

About