Key-Word-Aware Network for Referring Expression Image Segmentation

Shi, Hao; Li, Hongliang; Meng, Fanman; Wu, Qingbo

doi:10.1007/978-3-030-01231-1_3

Cited by 148 publications

(108 citation statements)

References 29 publications

Supporting

Mentioning

107

Contrasting

Order By: Relevance

“…It is difficult to directly exploit F to produce the segmentation output. In recent years, the attention mechanism [9,22,23,26,28] has been shown to be a powerful technique that can capture important information from raw features in either linguistic or visual representation. Different from above works, we propose a cross-modal self-attention module to jointly exploit attentions over multimodal features.…”

Section: Cross-modal Self-attentionmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Modal Self-Attention Network for Referring Image Segmentation

Rochan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

414

237

View full text Add to dashboard Cite

We consider the problem of referring image segmentation. Given an input image and a natural language expression, the goal is to segment the object referred by the language expression in the image. Existing works in this area treat the language expression and the input image separately in their representations. They do not sufficiently capture long-range correlations between these two modalities. In this paper, we propose a cross-modal self-attention (CMSA) module that effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the input image. In addition, we propose a gated multi-level fusion module to selectively integrate self-attentive cross-modal features corresponding to different levels in the image. This module controls the information flow of features at different levels. We validate the proposed approach on four evaluation datasets. Our proposed approach consistently outperforms existing state-of-the-art methods.

show abstract

Section: Cross-modal Self-attentionmentioning

confidence: 99%

“…A popular approach (e.g. [10,15,22]) in this area is to * Zhi Liu and Yang Wang are the corresponding authors Figure 1. (Best viewed in color) Illustration of our cross-modal self-attention mechanism.…”

Section: Introductionmentioning

confidence: 99%

Cross-Modal Self-Attention Network for Referring Image Segmentation

Rochan

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

414

237

View full text Add to dashboard Cite

show abstract

“…Their work uses users' click-through data. A new paradigm of attention-based mechanisms for referring expressions in image segmentation [30] is proposed which contains a keyword-aware network and query attention model that demonstrates the relationships with various image regions for a given query. Inspired by the idea of attention models, we modify this mechanism for patch alignments within images via information scent in the following section.…”

Section: Query Suggestion In Image Searchmentioning

confidence: 99%

“…Hence, finding useful patches for query expansion in an image based on textual queries (or descriptions) is the primary focus of our work. Past work [11,30] used both the query and image for typical retrieval and segmentation tasks. In our task formulation, we rely only upon a given arbitrary text prefix rather than having the entire text query which is used to perform search based on the image and supported by a modified deep language model [12] to find the most relevant patch in the image.…”

Section: Introductionmentioning

confidence: 99%

Utilising Information Foraging Theory for User Interaction with Image Query Auto-Completion

Jaiswal

Liu

Frommholz

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Query Auto-completion (QAC) is a prominently used feature in search engines, where user interaction with such explicit feature is facilitated by the possible automatic suggestion of queries based on a prefix typed by the user. Existing QAC models have pursued a little on user interaction and cannot capture a user's information need (IN) context. In this work, we devise a new task of QAC applied on an image for estimating patch (one of the key components of Information Foraging Theory) probabilities for query suggestion. Our work supports query completion by extending a user query prefix (one or two characters) to a complete query utilising a foraging-based probabilistic patch selection model. We present iBERT, to fine-tune the BERT (Bidirectional Encoder Representations from Transformers) model, which leverages combined textual-image queries for a solution to image QAC by computing probabilities of a large set of image patches. The reflected patch probabilities are used for selection while being agnostic to changing information need or contextual mechanisms. Experimental results show that query autocompletion using both natural language queries and images is more effective than using only language-level queries. Also, our fine-tuned iBERT model allows to efficiently rank patches in the image.

show abstract

“…Interactions between two modalities can associate each word with each image region, and further highlight the features of the target object for accurate segmentation. Recent works proposed to model the interactions by introducing unidirectional [8,9] or bi-directional [10] attention mechanisms.…”

Section: Introductionmentioning

confidence: 99%

Global Context Enhanced Multi-modal Fusion for Referring Image Segmentation

Yang

Huang

et al. 2020

Pattern Recognition and Computer Vision

View full text Add to dashboard Cite

In this work, we address the task of referring image segmentation (RIS), which aims at predicting a segmentation mask for the object described by a natural language expression. Most existing methods focus on establishing unidirectional or directional relationships between visual and linguistic features to associate two modalities together, while the multiscale context is ignored or insufficiently modeled. Multi-scale context is crucial to localize and segment those objects that have large scale variations during the multi-modal fusion process. To solve this problem, we propose a simple yet effective Cascaded Multi-modal Fusion (CMF) module, which stacks multiple atrous convolutional layers in parallel and further introduces a cascaded branch to fuse visual and linguistic features. The cascaded branch can progressively integrate multiscale contextual information and facilitate the alignment of two modalities during the multi-modal fusion process. Experimental results on four benchmark datasets demonstrate that our method outperforms most state-of-the-art methods. Code is available at https://github.com/jianhua2022/CMF-Refseg.

show abstract

Key-Word-Aware Network for Referring Expression Image Segmentation

Cited by 148 publications

References 29 publications

Cross-Modal Self-Attention Network for Referring Image Segmentation

Cross-Modal Self-Attention Network for Referring Image Segmentation

Utilising Information Foraging Theory for User Interaction with Image Query Auto-Completion

Global Context Enhanced Multi-modal Fusion for Referring Image Segmentation

Contact Info

Product

Resources

About