MaIL: A Unified Mask-Image-Language Trimodal Network for Referring Image Segmentation

Abstract

Referring image segmentation is a typical multi-modal task, which aims at generating a binary mask for referent described in given language expressions. Prior arts adopt a bimodal solution, taking images and languages as two modalities within an encoder-fusion-decoder pipeline. However, this pipeline is sub-optimal for the target task for two reasons. First, they only fuse high-level features produced by uni-modal encoders separately, which hinders sufficient cross-modal learning. Second, the uni-modal encoders are pre-trained independently, which brings inconsistency between pre-trained uni-modal tasks and the target multi-modal task. Besides, this pipeline often ignores or makes little use of intuitively beneficial instance-level features. To relieve these problems, we propose MaIL, which is a more concise encoder-decoder pipeline with a Mask-Image-Language trimodal encoder. Specifically, MaIL unifies uni-modal feature extractors and their fusion model into a deep modality interaction encoder, facilitating sufficient feature interaction across different modalities. Meanwhile, MaIL directly avoids the second limitation since no unimodal encoders are needed anymore. Moreover, for the first time, we propose to introduce instance masks as an additional modality, which explicitly intensifies instancelevel features and promotes finer segmentation results. The proposed MaIL set a new state-of-the-art on all frequentlyused referring image segmentation datasets, including Ref-COCO, RefCOCO+, and G-Ref, with significant gains, 3%-10% against previous best methods. Code will be released soon.

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

“…As shown in TABLE 7, the proposed approach outperforms MaIL and CRIS by around 2% ∼ 4% IoU without using large-scale vision-language datasets in pretraining, which demonstrates the effectiveness of our proposed modules with stronger visual and textual encoders. Especially, the proposed approach VLT achieves higher performance gain on more difficult dataset G-Ref that has a longer average sentence length and more complex and diverse word usages, e.g., VLT is ∼4% IoU better than MaIL [33] and LAVT [34] on test (U) of G-Ref. It demonstrates the proposed model's good ability in dealing with long and complex expressions with large diversities, which is mainly attributed to inputconditional query generation and selection that well cope with the diverse words/expressions, and masked contrastive learning that enhances the model's generalization ability.…”

“…We use the popular vision transformer backbone Swin-B [83] as visual encoder and BERT [47] as textual encoder to replace the Darknet53 [71] and bi-GRU [72], respectively. Methods pretrained on large-scale vision-language datasets are marked with †, e.g., MaIL [33] adopts ViLT [38] pre-trained on four large-scale vision-language pretraining datasets and CRIS [35] employs CLIP [40] pretrained on 400M image-text pairs. As shown in TABLE 7, the proposed approach outperforms MaIL and CRIS by around 2% ∼ 4% IoU without using large-scale vision-language datasets in pretraining, which demonstrates the effectiveness of our proposed modules with stronger visual and textual encoders.…”

“…Our concurrent work MDETR [31] employs DETR [15] to build an endto-end modulated detector and reason jointly over language and image. After the proposed VLT [32], transformer-based referring segmentation architectures receive more attention [33], [34], [35], [36], [37]. MaIL [33] follows the transformer architecture ViLT [38] and utilizes instance mask predicted by Mask R-CNN [39] as additional input.…”

“…After the proposed VLT [32], transformer-based referring segmentation architectures receive more attention [33], [34], [35], [36], [37]. MaIL [33] follows the transformer architecture ViLT [38] and utilizes instance mask predicted by Mask R-CNN [39] as additional input. Yang et al [34] propose Language-Aware Vision Transformer (LAVT) to conduct multi-modal fusion at intermediate levels of the network.…”

Learning Aligned Cross-modal Representations for Referring Image Segmentation

“…Decoder-fusion methods only interact with high-level features of two modalities and miss the importance of low-level visual features such as texture, shape, and color, as well as word features. Some recent works [10,23,41] proposed encoder-fusion methods to solve the referring image segmentation problem. These methods perform cross-modal interactions on visual and linguistic features in the early encoder stage to update visual features.…”