Modeling Context in Referring Expressions

Yu, Licheng; Poirson, Patrick; Yang, Shan; Berg, Alexander C.; Berg, Tamara L.

doi:10.48550/arxiv.1608.00272

Cited by 29 publications

(3 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we solve a much harder task -we train our model to directly predict the bounding box, given a referring expression and the associated image. There are three established datasets for this task called RefCOCO, RefCOCO+ [71] and Ref-COCOg [36]. Since during pre-training we annotate every object referred to within the text, there is a slight shift in the way the model is used in this task.…”

Section: Downstream Tasksmentioning

confidence: 99%

“…We hypothesize that it enables the attention pattern for each question type to specialize accordingly to the task, there-by yielding better performance. C. Dataset constructions MS COCO On the COCO dataset, we include annotations from the referring expressions datasets (RefCOCO [71], Ref-COCO+ [71] and RefCOCOg [36] datasets). By construction, in this dataset, each referring expression is a whole sentence that describes one object in the image, where the constituent noun phrases from the sentences are not themselves annotated.…”

Section: B32 Question Answering Ablationsmentioning

confidence: 99%

See 1 more Smart Citation

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

Kamath¹,

Singh²,

LeCun³

et al. 2021

Preprint

View full text Add to dashboard Cite

Multi-modal reasoning systems rely on a pre-trained object detector to extract regions of interest from the image. However, this crucial module is typically used as a black box, trained independently of the downstream task and on a fixed vocabulary of objects and attributes. This makes it challenging for such systems to capture the long tail of visual concepts expressed in free form text. In this paper we propose MDETR, an end-to-end modulated detector that detects objects in an image conditioned on a raw text query, like a caption or a question. We use a transformer-based architecture to reason jointly over text and image by fusing the two modalities at an early stage of the model. We pre-train the network on 1.3M text-image pairs, mined from pre-existing multi-modal datasets having explicit alignment between phrases in text and objects in the image. We then fine-tune on several downstream tasks such as phrase grounding, referring expression comprehension and segmentation, achieving state-of-the-art results on popular benchmarks. We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting. We show that our pre-training approach provides a way to handle the long tail of object categories which have very few labelled instances. Our approach can be easily extended for visual question answering, achieving competitive performance on GQA and CLEVR. The code and models are available at https://github.com/ashkamath/mdetr.

show abstract

Section: Downstream Tasksmentioning

confidence: 99%

Section: B32 Question Answering Ablationsmentioning

confidence: 99%

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

Kamath¹,

Singh²,

LeCun³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For further insight on this case we examine the RMAE-coverage curve (Figure 3), where we see that while combined replacement reaches a lower RMAE, it does so slowly. This is likely due to the difference in data present in the two splits: TestA focuses primarily on people, while TestB focuses on objects (Yu et al 2016). The more varied set of classes leads to a more spread output distribution, meaning a target object may be re-queried several times before the combined distribution achieves a high certainty.…”

Section: Smart Replacementmentioning

confidence: 99%

Evaluating and Improving Interactions with Hazy Oracles

Lemmer¹,

Corso²

2021

Preprint

View full text Add to dashboard Cite

To build a shared perception of the world, humans rely on the ability to resolve misunderstandings by requesting and accepting clarifications. However, when evaluating visiolinguistic models, metrics such as accuracy enforce the assumption that a decision must be made based on a single piece of evidence. In this work, we relax this assumption for the task of referring expression comprehension by allowing the model to request help when its confidence is low. We consider two ways in which this help can be provided: multimodal requery, where the user is allowed to point or click to provide additional information to the model, and rephrase re-query, where the user is only allowed to provide another referring expression. We demonstrate the importance of re-query by showing that providing the best referring expression for all objects can increase accuracy by up to 21.9% and that this accuracy can be matched by re-querying only 12% of initial referring expressions. We further evaluate re-query functions for both multimodal and rephrase re-query across three modern approaches and demonstrate combined replacement for rephrase re-query, which improves average single-query performance by up to 6.5% and converges to as close as 1.6% of the upper bound of single-query performance.

show abstract

Using Syntax to Ground Referring Expressions in Natural Images

Cirik

Berg-Kirkpatrick

Morency

2018

AAAI

View full text Add to dashboard Cite

We introduce GroundNet, a neural network for referring expression recognition---the task of localizing (or grounding) in an image the object referred to by a natural language expression. Our approach to this task is the first to rely on a syntactic analysis of the input referring expression in order to inform the structure of the computation graph. Given a parse tree for an input expression, we explicitly map the syntactic constituents and relationships present in the tree to a composed graph of neural modules that defines our architecture for performing localization. This syntax-based approach aids localization of both the target object and auxiliary supporting objects mentioned in the expression. As a result, GroundNet is more interpretable than previous methods: we can (1) determine which phrase of the referring expression points to which object in the image and (2) track how the localization of the target object is determined by the network. We study this property empirically by introducing a new set of annotations on the GoogleRef dataset to evaluate localization of supporting objects. Our experiments show that GroundNet achieves state-of-the-art accuracy in identifying supporting objects, while maintaining comparable performance in the localization of target objects.

show abstract

Modeling Context in Referring Expressions

Cited by 29 publications

References 28 publications

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

Evaluating and Improving Interactions with Hazy Oracles

Using Syntax to Ground Referring Expressions in Natural Images

Contact Info

Product

Resources

About