Localized Vision-Language Matching for Open-vocabulary Object Detection

Bravo, María Alejandra; Mittal, Sudhanshu; Brox, Thomas

doi:10.1007/978-3-031-16788-1_24

Cited by 12 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…VL-PLM [215] proposes to train Faster R-CNN as a two-stage class-agnostic proposal generator using a detection dataset without category information. LocOV [238] uses classagnostic proposals in RPN to train Faster R-CNN by matching the region features and word embeddings from image and caption, respectively. From the data generation view, several works [138], [239], [240] adopt the diffusion model to generate the on-target data for effective training.…”

Section: Open Vocabulary Object Detectionmentioning

confidence: 99%

Towards Open Vocabulary Learning: A Survey

Wu,

Li,

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

In the field of visual scene understanding, deep neural networks have made impressive advancements in various core tasks like segmentation, tracking, and detection. However, most approaches operate on the close-set assumption, meaning that the model can only identify pre-defined categories that are present in the training set. Recently, open vocabulary settings were proposed due to the rapid progress of vision language pre-training. These new approaches seek to locate and recognize categories beyond the annotated label space. The open vocabulary approach is more general, practical, and effective than weakly supervised and zero-shot settings. This paper thoroughly reviews open vocabulary learning, summarizing and analyzing recent developments in the field. In particular, we begin by juxtaposing open vocabulary learning with analogous concepts such as zero-shot learning, open-set recognition, and out-of-distribution detection. Subsequently, we examine several pertinent tasks within the realms of segmentation and detection, encompassing long-tail problems, few-shot, and zero-shot settings. As a foundation for our method survey, we first elucidate the fundamental principles of detection and segmentation in close-set scenarios. Next, we examine various contexts where open vocabulary learning is employed, pinpointing recurring design elements and central themes. This is followed by a comparative analysis of recent detection and segmentation methodologies in commonly used datasets and benchmarks. Our review culminates with a synthesis of insights, challenges, and discourse on prospective research trajectories. To our knowledge, this constitutes the inaugural exhaustive literature review on open vocabulary learning. We keep tracing related works at https://github.com/jianzongwu/Awesome-Open-Vocabulary.

show abstract

Section: Open Vocabulary Object Detectionmentioning

confidence: 99%

Towards Open Vocabulary Learning: A Survey

Wu,

Li,

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…We utilize the OVAD dataset [41] and the VAW dataset [42] to obtain image-text pairs associated with attribute knowledge. The OVAD dataset consists of 80 object classes and 117 attribute classes.…”

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

Pro-MA: Progressively Margin-based Attribution in Pre-trained Vision-Language Models

You,

2024

Preprint

View full text Add to dashboard Cite

Knowledge attribution is focused on analyzing the internal knowledge architecture of neural networks, aiming to accurately identify the factual knowledge stored within a model. Existing knowledge attribution methods concentrate on Pre-trained Language Models (PLMs), with limited exploration in Pre-trained Vision-Language Models (VLMs). However, the knowledge attribution methods of PLMs cannot be directly utilized on VLMs due to different decision-making processes and unstable integral gradients. Specifically, VLMs involve extra knowledge-irrelevant steps like localization and show high variance in gradient distributions along the attribution paths. To address the above challenges, we propose Progressive Margin-based Attribution (Pro-MA) to achieve more accurate attribute knowledge attribution in VLMs. Firstly, we employ cross-attribute comparison to identify and eliminate irrelevant knowledge by constructing high-similarity negative samples. Secondly, we utilize progressive attribution scoring that quantifies the contribution of neurons to the output by the margin of the attribute-value loss function along the path of active neurons. Evaluation on the OVAD and VAW datasets shows that Pro-MA outperforms the state-of-the-art knowledge attribution methods.

show abstract

“…As the rising of image-text pair pre-training multimodal models like CLIP (Radford et al, 2021), ALIGN (Jia et al, 2021), ALBEF , researchers spend effort in combining cross-modal knowledge with dense prediction tasks like detection Gu et al, 2021;Bravo et al, 2023; and segmentation (Kim et al, 2023;Liang et al, 2022;Rao et al, 2021;Xu et al, 2023d;Qin et al, 2023;Liang et al, 2023;Xu et al, 2023a,b). However, the text used in these works is restricted to object class words or attributes (Li et al, 2022a).…”

Section: Zero-shot Segmentationmentioning

confidence: 99%

Text Augmented Spatial Aware Zero-shot Referring Image Segmentation

Suo,

Zhu,

Yang

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

In this paper, we study a challenging task of zero-shot referring image segmentation. This task aims to identify the instance mask that is most related to a referring expression without training on pixel-level annotations. Previous research takes advantage of pre-trained crossmodal models, e.g., CLIP, to align instancelevel masks with referring expressions. Yet, CLIP only considers the global-level alignment of image-text pairs, neglecting fine-grained matching between the referring sentence and local image regions. To address this challenge, we introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework that is training-free and robust to various visual encoders. TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial rectifier for mask post-processing. Notably, the text-augmented visual-text matching score leverages a P -score and an N -score in addition to the typical visual-text matching score. The P -score is utilized to close the visual-text domain gap through a surrogate captioning model, where the score is computed between the surrogate model-generated texts and the referring expression. The N -score considers the fine-grained alignment of region-text pairs via negative phrase mining, encouraging the masked image to be repelled from the mined distracting phrases. Extensive experiments are conducted on various datasets, including Re-fCOCO, RefCOCO+, and RefCOCOg. The proposed method clearly outperforms state-ofthe-art zero-shot referring image segmentation methods.

show abstract

Localized Vision-Language Matching for Open-vocabulary Object Detection

Cited by 12 publications

References 29 publications

Towards Open Vocabulary Learning: A Survey

Towards Open Vocabulary Learning: A Survey

Pro-MA: Progressively Margin-based Attribution in Pre-trained Vision-Language Models

Text Augmented Spatial Aware Zero-shot Referring Image Segmentation

Contact Info

Product

Resources

About