Open-Vocabulary Panoptic Segmentation with MaskCLIP

Zheng, Dong; Wang, Jieke; Tu, Zhuowen

doi:10.48550/arxiv.2208.08984

Cited by 6 publications

(23 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, the mask generator is CLIP-unaware, further limiting its performance. MaskCLIP [10] improves the two-stage framework by progressively refining the predicted masks by the CLIP encoder, and applying masks in attention layers to avoid forwarding multiple times, which was first introduced by [7]. However, MaskCLIP still needs a heavy mask generator, the initial mask prediction is also CLIP-unaware, and the mask prediction and recognition are coupled.…”

Section: Related Workmentioning

confidence: 99%

“…Our approach is an end-to-end framework, the mask prediction is lightweight and CLIP-aware, and the mask recognition is decoupled from mask prediction. These differences allow our approach can better leverage the capability of CLIP than two-stage approaches [10,33].…”

Section: Related Workmentioning

confidence: 99%

“…Furthermore, we compare with other methods: Sim-Seg [33], OvSeg [22], and MaskCLIP [10], which also use CLIP ViT models, in terms of trainable parameters, GFLOPs and inference time (FPS). For a fair comparison, we test all methods under the same environment: single Titan Xp GPU, Xeon E5 v2 CPU (32 core), 252G RAM, Py-Torch 1.9.0, and CUDA 11.3.…”

Section: System Level Comparisonmentioning

confidence: 99%

“…Along with the excellent performance, our approach requires only 8.4M trainable parameters with 64.3 GFLOPs, which is only 13% and 20% of [10], 6% and less than 1% of [22], respectively.…”

Section: Introductionmentioning

confidence: 97%

See 3 more Smart Citations

Side Adapter Network for Open-Vocabulary Semantic Segmentation

Xu¹,

Zhang²,

Wei³

et al. 2023

Preprint

View full text Add to dashboard Cite

This paper presents a new framework for openvocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, it can be very light. In addition, the entire network can be trained end-to-end, allowing the side network to be adapted to the frozen CLIP model, which makes the predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only adds a few additional trainable parameters. We evaluate our approach on multiple semantic segmentation benchmarks. Our method significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed. We hope our approach will serve as a solid baseline and help ease future research in open-vocabulary semantic segmentation. The code will be available at https://github.com/MendelXu/SAN.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: System Level Comparisonmentioning

confidence: 99%

“…Along with the excellent performance, our approach requires only 8.4M trainable parameters with 64.3 GFLOPs, which is only 13% and 20% of [10], 6% and less than 1% of [22], respectively.…”

Section: Introductionmentioning

confidence: 97%

See 2 more Smart Citations

Side Adapter Network for Open-Vocabulary Semantic Segmentation

Xu¹,

Zhang²,

Wei³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…• CLIP-based Segmentation. Many segmentation models directly adapt the pre-trained CLIP to pixel-level visual recognition tasks, including PhraseCut (Wu et al, 2020), OpenSeg (Ghiasi et al, 2022), CLIPSeg (Lüddecke and Ecker, 2022), ZS-Seg (Xu et al, 2021d), MaskCLIP (Zhou et al, 2022a), DenseCLIP (Rao et al, 2021) and MaskCLIP (Ding et al, 2022b). OpenSeg (Ghiasi et al, 2022) also performs model learning with class agnostic mask annotations for generating mask proposals.…”

Section: Vlp For Segmentationmentioning

confidence: 99%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Gan¹,

Fu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.♠ Zhe Gan and Jianfeng Gao initiated the project. Zhe Gan and Linjie Li took lead in the writing of Chapter 1. Linjie Li and Jianfeng Gao took lead in the writing of Chapter 2. Zhe Gan further took lead in the writing of Chapter 3 and 7. Chunyuan Li took lead in the writing of Chapter 4. Linjie Li further took lead in the writing of Chapter 5. Lijuan Wang and Zicheng Liu took lead in the writing of Chapter 6. All the authors provided project advice, and contributed to paper editing and proofreading.

show abstract

Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition

Wang,

Xie,

Wang

et al. 2023

Proceedings of the 31st ACM International Conference on Multimedia

View full text Add to dashboard Cite

In this paper, we explore the potential of the Contrastive Language-Image Pretraining (CLIP) model in scene text recognition (STR), and establish a novel Symmetrical Linguistic Feature Distillation framework (named CLIP-OCR) to leverage both visual and linguistic knowledge in CLIP. Different from previous CLIP-based methods mainly considering feature generalization on visual encoding, we propose a symmetrical distillation strategy (SDS) that further captures the linguistic knowledge in the CLIP text encoder. By cascading the CLIP image encoder with the reversed CLIP text encoder, a symmetrical structure is built with an image-to-text feature flow that covers not only visual but also linguistic information for distillation. Benefiting from the natural alignment in CLIP, such guidance flow provides a progressive optimization objective from vision to language, which can supervise the STR feature forwarding process layer-by-layer. Besides, a new Linguistic Consistency Loss (LCL) is proposed to enhance the linguistic capability by considering second-order statistics during the optimization. Overall, CLIP-OCR is the first to design a smooth transition between image and text for the STR task. Extensive experiments demonstrate the effectiveness of CLIP-OCR with 93.8% average accuracy on six popular STR benchmarks. Code will be available at https://github.com/wzx99/CLIPOCR. CCS CONCEPTS• Applied computing → Optical character recognition.

show abstract

Open-Vocabulary Panoptic Segmentation with MaskCLIP

Cited by 6 publications

References 26 publications

Side Adapter Network for Open-Vocabulary Semantic Segmentation

Side Adapter Network for Open-Vocabulary Semantic Segmentation

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition

Contact Info

Product

Resources

About