Improving Referring Expression Grounding With Cross-Modal Attention-Guided Erasing

Liu, Xihui; Wang, Zihao; Shao, Jing; Wang, Xiaogang; Li, Hongsheng

doi:10.1109/cvpr.2019.00205

Cited by 167 publications

(99 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In future work we plan to replicate the results reported by Krishna et al (2018) and to compare it with our object-only baseline. We hope to do the same for other published results on referring relationships using the VRD dataset, among other datasets (Cirik et al, 2018a;Liu et al, 2019;Raboh et al, 2019).…”

Section: Revisiting "Referring Relationship" Groundingmentioning

confidence: 55%

Revisiting Visual Grounding

Conser¹,

Hahn²,

Watson³

et al. 2019

Proceedings of the Second Workshop on Shortcomings in Vision and Language

View full text Add to dashboard Cite

We revisit a particular visual grounding method: the "Image Retrieval Using Scene Graphs" (IRSG) system of Johnson et al. (2015). Our experiments indicate that the system does not effectively use its learned objectrelationship models. We also look closely at the IRSG dataset, as well as the widely used Visual Relationship Dataset (VRD) that is adapted from it. We find that these datasets exhibit biases that allow methods that ignore relationships to perform relatively well. We also describe several other problems with the IRSG dataset, and report on experiments using a subset of the dataset in which the biases and other problems are removed. Our studies contribute to a more general effort: that of better understanding what machine learning methods that combine language and vision actually learn and what popular datasets actually test.

show abstract

Section: Revisiting "Referring Relationship" Groundingmentioning

confidence: 55%

Revisiting Visual Grounding

Conser¹,

Hahn²,

Watson³

et al. 2019

Proceedings of the Second Workshop on Shortcomings in Vision and Language

View full text Add to dashboard Cite

show abstract

“…The first experiment compared the performance of our method and several baseline models in matching detection task. Based on CMNs [23], MAttNet [24], and CM‐Att [46], we propose the baseline models for matching detection called CMNS‐baseline, MAttNet‐baseline, and CM‐Att‐baseline. The baseline models determine whether the expression matches the image by setting a threshold in its predicted highest score.…”

Section: Methodsmentioning

confidence: 99%

Referring expression comprehension model with matching detection and linguistic feedback

Wang

Cui

Liu

et al. 2020

IET Computer Vision

View full text Add to dashboard Cite

The task of referring expression comprehension (REC) is to localise an image region of a specific object described by a natural language expression, and all existing REC methods assume that the object described by the referring expression must be located in the given image. However, this assumption is not correct in some real applications. For example, a visually impaired user might tell his robot ‘please take the laptop on the table to me’. In fact, the laptop is not on the table anymore. To address this problem, the authors propose a novel REC model to deal with the situation where expression‐image mismatching occurs and explain the mismatching by linguistic feedback. The authors' REC model consists of four modules: the expression parsing module, the entity detection module, the relationship detection module, and the matching detection module. They built a data set called NP‐RefCOCO+ from RefCOCO+ including both positive samples and negative samples. The positive samples are original expression‐image pairs in RefCOCO+. The negative samples are the expression‐image pairs in RefCOCO+, whose expressions are replaced. They evaluate the model on NP‐RefCOCO+ and the experimental results show the advantages of their method for dealing with the problem of expression‐image mismatching.

show abstract

“…Many tasks [1,2,3,4,5,6] have been explored to link the relation between vision and language. Among these tasks, image-text matching [1,3] aims to learn the semantic similarities between images and sentences, which can be applied to bi-directional image and text retrieval tasks that retrieve images given sentences or retrieve sentences for a query image.…”

Section: Introductionmentioning

confidence: 99%

Attend, Correct And Focus: A Bidirectional Correct Attention Network For Image-Text Matching

Liu¹,

Wang²,

Meng

et al. 2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Image-text matching task aims to learn the fine-grained correspondences between images and sentences. Existing methods use attention mechanism to learn the correspondences by attending to all fragments without considering the relationship between fragments and global semantics, which inevitably lead to semantic misalignment among irrelevant fragments. To this end, we propose a Bidirectional Correct Attention Network (BCAN), which leverages global similarities and local similarities to reassign the attention weight, to avoid such semantic misalignment. Specifically, we introduce a global correct unit to correct the attention focused on relevant fragments in irrelevant semantics. A local correct unit is used to correct the attention focused on irrelevant fragments in relevant semantics. Experiments on Flickr30K and MSCOCO datasets verify the effectiveness of our proposed BCAN by outperforming both previous attention-based methods and state-of-the-art methods. Code can be found at: https://github.com/liuyyy111/BCAN.

show abstract

Improving Referring Expression Grounding With Cross-Modal Attention-Guided Erasing

Cited by 167 publications

References 42 publications

Revisiting Visual Grounding

Revisiting Visual Grounding

Referring expression comprehension model with matching detection and linguistic feedback

Attend, Correct And Focus: A Bidirectional Correct Attention Network For Image-Text Matching

Contact Info

Product

Resources

About