LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

Abstract

Referring image segmentation segments an image from a language expression. With the aim of producing high-quality masks, existing methods often adopt iterative learning approaches that rely on RNNs or stacked attention layers to refine vision-language features. Despite their complexity, RNN-based methods are subject to specific encoder choices, while attention-based methods offer limited gains. In this work, we introduce a simple yet effective alternative for progressively learning discriminative multi-modal features. The core idea of our approach is to leverage a continuously updated query as the representation of the target object and at each iteration, strengthen multi-modal features strongly correlated to the query while weakening less related ones. As the query is initialized by language features and successively updated by object features, our algorithm gradually shifts from being localization-centric to segmentation-centric. This strategy enables the incremental recovery of missing object parts and/or removal of extraneous parts through iteration. Compared to its counterparts, our method is more versatile-it can be plugged into prior arts straightforwardly and consistently bring improvements. Experimental results on the challenging datasets of RefCOCO, RefCOCO+, and G-Ref demonstrate its advantage with respect to the state-of-the-art methods.

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

“…As shown in TABLE 7, the proposed approach outperforms MaIL and CRIS by around 2% ∼ 4% IoU without using large-scale vision-language datasets in pretraining, which demonstrates the effectiveness of our proposed modules with stronger visual and textual encoders. Especially, the proposed approach VLT achieves higher performance gain on more difficult dataset G-Ref that has a longer average sentence length and more complex and diverse word usages, e.g., VLT is ∼4% IoU better than MaIL [33] and LAVT [34] on test (U) of G-Ref. It demonstrates the proposed model's good ability in dealing with long and complex expressions with large diversities, which is mainly attributed to inputconditional query generation and selection that well cope with the diverse words/expressions, and masked contrastive learning that enhances the model's generalization ability.…”

“…Our concurrent work MDETR [31] employs DETR [15] to build an endto-end modulated detector and reason jointly over language and image. After the proposed VLT [32], transformer-based referring segmentation architectures receive more attention [33], [34], [35], [36], [37]. MaIL [33] follows the transformer architecture ViLT [38] and utilizes instance mask predicted by Mask R-CNN [39] as additional input.…”

“…MaIL [33] follows the transformer architecture ViLT [38] and utilizes instance mask predicted by Mask R-CNN [39] as additional input. Yang et al [34] propose Language-Aware Vision Transformer (LAVT) to conduct multi-modal fusion at intermediate levels of the network. CRIS [35] employs CLIP [40] pretrained on 400M image text pairs and transfers CLIP from text-to-image matching to text-to-pixel matching.…”

Learning Aligned Cross-modal Representations for Referring Image Segmentation

“…Recent studies [10,19,41] have shown that cross-modal interactions during feature extraction can further enhance multi-modal alignment. Feng et al [10] replaced the vision encoder with a multi-modal encoder by adopting an early cross-modal interaction strategy, which achieved deep interweaving between visual and linguistic features.…”

“…Decoder-fusion methods only interact with high-level features of two modalities and miss the importance of low-level visual features such as texture, shape, and color, as well as word features. Some recent works [10,23,41] proposed encoder-fusion methods to solve the referring image segmentation problem. These methods perform cross-modal interactions on visual and linguistic features in the early encoder stage to update visual features.…”