ReSTR: Convolution-free Referring Image Segmentation Using Transformers

Kim, Namyup; Kim, Dong-Won; Kwak, Suha; Lan, Cuiling; Zeng, Wenjun

doi:10.1109/cvpr52688.2022.01761

Cited by 97 publications

(18 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our concurrent work MDETR [31] employs DETR [15] to build an endto-end modulated detector and reason jointly over language and image. After the proposed VLT [32], transformer-based referring segmentation architectures receive more attention [33], [34], [35], [36], [37]. MaIL [33] follows the transformer architecture ViLT [38] and utilizes instance mask predicted by Mask R-CNN [39] as additional input.…”

Section: Referring Segmentationmentioning

confidence: 99%

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Ding

Liu

Wang

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different ways to understand the dynamic emphasis of a language expression, especially when interacting with the image. However, the learned queries in existing transformer works are fixed after training, which cannot cope with the randomness and huge diversity of the language expressions. To address this issue, we propose a Query Generation Module, which dynamically produces multiple sets of input-specific queries to represent the diverse comprehensions of language expression. To find the best among these diverse comprehensions, so as to generate a better mask, we propose a Query Balance Module to selectively fuse the corresponding responses of the set of queries. Furthermore, to enhance the model's ability in dealing with diverse language expressions, we consider inter-sample learning to explicitly endow the model with knowledge of understanding different language expressions to the same object. We introduce masked contrastive learning to narrow down the features of different expressions for the same target object while distinguishing the features of different objects. The proposed approach is lightweight and achieves new state-of-the-art referring segmentation results consistently on five datasets.

show abstract

Section: Referring Segmentationmentioning

confidence: 99%

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Ding

Liu

Wang

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Following the unprecedented success in natural language tasks, transformers [48] have also made great achievements in image recognition tasks recently. Te ViT model has become very popular in various computer vision tasks including image classifcation [16], image detection [49], image segmentation [50], and so on. In the feld of natural image recognition, ViT and its derived instances have achieved state-of-the-art performance on several benchmark datasets.…”

Section: Visual Transformer In Medicinementioning

confidence: 99%

Classification of Diabetic Retinopathy Severity in Fundus Images Using the Vision Transformer and Residual Attention

Wang

et al. 2023

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Diabetic retinopathy (DR) is a common retinal vascular disease, which can cause severe visual impairment. It is of great clinical significance to use fundus images for intelligent diagnosis of DR. In this paper, an intelligent DR classification model of fundus images is proposed. This method can detect all the five stages of DR, including of no DR, mild, moderate, severe, and proliferative. This model is composed of two key modules. FEB, feature extraction block, is mainly used for feature extraction of fundus images, and GPB, grading prediction block, is used to classify the five stages of DR. The transformer in the FEB has more fine-grained attention that can pay more attention to retinal hemorrhage and exudate areas. The residual attention in the GPB can effectively capture different spatial regions occupied by different classes of objects. Comprehensive experiments on DDR datasets well demonstrate the superiority of our method, and compared with the benchmark method, our method has achieved competitive performance.

show abstract

“…Based on this pipeline, some works [2,15,25,42] improved the performance by utilizing more powerful feature encoders and designing more ingenious fusion strategies (e.g., recurrent fusion). With the success of attention mechanisms in the communities of natural language processing and computer vision, follow-up works [8,21,29,43] attempted to model intra-modal and cross-modal relationships by adopting self-attention and cross-attention operations. For example, Ye et al [43] proposed Cross-Modal Self-Attention (CMSA) to highlight informative visual and linguistic elements.…”

Section: Related Workmentioning

confidence: 99%

Learning Aligned Cross-modal Representations for Referring Image Segmentation

Zhou¹,

Chen²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Referring image segmentation aims to segment the image region of interest according to the given language expression, which is a typical multi-modal task. One of the critical challenges of this task is to align semantic representations for different modalities including vision and language. To achieve this, previous methods perform cross-modal interactions to update visual features but ignore the role of integrating fine-grained visual features into linguistic features. We present AlignFormer, an end-to-end framework for referring image segmentation. Our AlignFormer views the linguistic feature as the center embedding and segments the region of interest by pixels grouping based on the center embedding. For achieving the pixel-text alignment, we design a Vision-Language Bidirectional Attention module (VLBA) and resort contrastive learning. Concretely, the VLBA enhances visual features by propagating semantic text representations to each pixel and promotes linguistic features by fusing fine-grained image features. Moreover, we introduce the cross-modal instance contrastive loss to alleviate the influence of pixel samples in ambiguous regions and improve the ability to align multi-modal representations. Extensive experiments demonstrate that our Align-Former achieves a new state-of-the-art performance on Re-fCOCO, RefCOCO+, and RefCOCOg by large margins.

show abstract

ReSTR: Convolution-free Referring Image Segmentation Using Transformers

Cited by 97 publications

References 83 publications

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Classification of Diabetic Retinopathy Severity in Fundus Images Using the Vision Transformer and Residual Attention

Learning Aligned Cross-modal Representations for Referring Image Segmentation

Contact Info

Product

Resources

About