PIRC Net: Using Proposal Indexing, Relationships and Context for Phrase Grounding

Kovvuri, Rama; Nevatia, Ram

doi:10.1007/978-3-030-20870-7_28

Cited by 11 publications

(5 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the model can not recover when the region proposal network fails to return a bounding box for the object of interest. In this case, relative regression or query-based approaches can be used [7,22]. Backbones and word attention Surprisingly, some of the higher ranked methods use the same visual backbone as their lower ranked competitors.…”

Section: Qualitative Comparisonmentioning

confidence: 99%

See 1 more Smart Citation

Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

Deruyttere

Vandenhende

Grujicic

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The task of visual grounding requires locating the most relevant region or object in an image, given a natural language query. So far, progress on this task was mostly measured on curated datasets, which are not always representative of human spoken language. In this work, we deviate from recent, popular task settings and consider the problem under an autonomous vehicle scenario. In particular, we consider a situation where passengers can give free-form natural language commands to a vehicle which can be associated with an object in the street scene. To stimulate research on this topic, we have organized the Commands for Autonomous Vehicles (C4AV) challenge based on the recent Talk2Car dataset. This paper presents the results of the challenge. First, we compare the used benchmark against existing datasets for visual grounding. Second, we identify the aspects that render top-performing models successful, and relate them to existing state-of-the-art models for visual grounding, in addition to detecting potential failure cases by evaluating on carefully selected subsets. Finally, we discuss several possibilities for future work.

show abstract

Section: Qualitative Comparisonmentioning

confidence: 99%

Section: Qualitative Comparisonmentioning

confidence: 99%

Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

Deruyttere

Vandenhende

Grujicic

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Hence, the ground truth has better precision in this dataset. Our model achieves 12.30% better R@1 performance than the current state-of-the-art one-stage [22] and two-stage [26] approaches. This performance boost can be attributed to the following key points: (1).…”

Section: Quantitative Analysismentioning

confidence: 84%

“…In this section, we present experiments to evaluate our proposed model MAGNet on varieties of datasets with multiple evaluation metrics and compare our results to the state-of-the-art visual grounding methods [26] and [27]. Results of ablation studies with different configurations will also be reported to further explain the design decisions of the proposed model.…”

Section: Methodsmentioning

confidence: 99%

MAGNet: Multi-Region Attention-Assisted Grounding of Natural Language Queries at Phrase Level

Shrestha

Pugdeethosapol

Fang

et al. 2020

Preprint

View full text Add to dashboard Cite

Grounding free-form textual queries necessitates an understanding of these textual phrases and its relation to the visual cues to reliably reason about the described locations. Spatial attention networks are known to learn this relationship and focus its gaze on salient objects in the image. Thus, we propose to utilize spatial attention networks for image-level visual-textual fusion preserving local (word) and global (phrase) information to refine region proposals with an in-network Region Proposal Network (RPN) and detect single or multiple regions for a phrase query. We focus only on the phrase query -ground truth pair (referring expression) for a model independent of the constraints of the datasets i.e. additional attributes, context etc. For such referring expression dataset ReferIt game, our Multi-region Attention-assisted Grounding network (MAGNet) achieves over 12% improvement over the state-of-the-art. Without the context from image captions and attribute information in Flickr30k Entities, we still achieve competitive results compared to the state-of-the-art.

show abstract

“…Flickr30K accuracy (%) Similarity Net [54] ResNet-101 60.89 CITE [24] ResNet-101 61.33 PIRC [55] ResNet-101 72.83 DDPN [56] ResNet-101 73.30 LCMCG [57] ResNet-101 76.74 ZSGNet [58] ResNet-50 63.39 FAOA [32] DarkNet-53 68.71 ReSC-Large [50] DarkNet-53 69.28 TransVG [53] ResNet-50 78.47 TransVG [53] ResNet-101 79.…”

Section: Models Backbonementioning

confidence: 99%

Multi-modal interaction with transformers: bridging robots and human with natural language

Wang,

Zhou,

et al. 2023

Robotica

View full text Add to dashboard Cite

The language-guided visual robotic grasping task focuses on enabling robots to grasp objects based on human language instructions. However, real-world human-robot collaboration tasks often involve situations with ambiguous language instructions and complex scenarios. These challenges arise in the understanding of linguistic queries, discrimination of key concepts in visual and language information, and generation of executable grasping configurations for the robot’s end-effector. To overcome these challenges, we propose a novel multi-modal transformer-based framework in this study, which assists robots in localizing spatial interactions of objects using text queries and visual sensing. This framework facilitates object grasping in accordance with human instructions. Our developed framework consists of two main components. First, a visual-linguistic transformer encoder is employed to model multi-modal interactions for objects referred to in the text. Second, the framework performs joint spatial localization and grasping. Extensive ablation studies have been conducted on multiple datasets to evaluate the advantages of each component in our model. Additionally, physical experiments have been performed with natural language-driven human-robot interactions on a physical robot to validate the practicality of our approach.

show abstract

PIRC Net: Using Proposal Indexing, Relationships and Context for Phrase Grounding

Cited by 11 publications

References 31 publications

Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

MAGNet: Multi-Region Attention-Assisted Grounding of Natural Language Queries at Phrase Level

Multi-modal interaction with transformers: bridging robots and human with natural language

Contact Info

Product

Resources

About