Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation

Jianzong, Wu,; Li, Xiangtai; Ding, Henghui; Loy, Chen Change

doi:10.36227/techrxiv.22082723

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It also proposes to use cached web data to enhance the novel classes during training. Based on the DETR [4] framework, CORA [219] proposes region prompting and anchor pre-matching. The former reduces the gap between the whole image and region distributions by prompting the region features of the CLIP-based region classifier, while the latter learns generalizable object localization via a class-aware matching mechanism.…”

Section: Open Vocabulary Object Detectionmentioning

confidence: 99%

Towards Open Vocabulary Learning: A Survey

Wu,

Li,

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

In the field of visual scene understanding, deep neural networks have made impressive advancements in various core tasks like segmentation, tracking, and detection. However, most approaches operate on the close-set assumption, meaning that the model can only identify pre-defined categories that are present in the training set. Recently, open vocabulary settings were proposed due to the rapid progress of vision language pre-training. These new approaches seek to locate and recognize categories beyond the annotated label space. The open vocabulary approach is more general, practical, and effective than weakly supervised and zero-shot settings. This paper thoroughly reviews open vocabulary learning, summarizing and analyzing recent developments in the field. In particular, we begin by juxtaposing open vocabulary learning with analogous concepts such as zero-shot learning, open-set recognition, and out-of-distribution detection. Subsequently, we examine several pertinent tasks within the realms of segmentation and detection, encompassing long-tail problems, few-shot, and zero-shot settings. As a foundation for our method survey, we first elucidate the fundamental principles of detection and segmentation in close-set scenarios. Next, we examine various contexts where open vocabulary learning is employed, pinpointing recurring design elements and central themes. This is followed by a comparative analysis of recent detection and segmentation methodologies in commonly used datasets and benchmarks. Our review culminates with a synthesis of insights, challenges, and discourse on prospective research trajectories. To our knowledge, this constitutes the inaugural exhaustive literature review on open vocabulary learning. We keep tracing related works at https://github.com/jianzongwu/Awesome-Open-Vocabulary.

show abstract

Section: Open Vocabulary Object Detectionmentioning

confidence: 99%

Towards Open Vocabulary Learning: A Survey

Wu,

Li,

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Rahman et al [32] further advanced ZSD with an enhanced visual-semantic alignment technique, employing a polarity loss function to improve discrimination between positive and negative predictions significantly. More recently, the advent of vision-language pretraining has led to ZSD being conceptualized as an image-text matching problem [33]- [35], leveraging large-scale imagetext data to expand the number of training classes. Inspired by these methods, this research utilizes a pre-trained visionlanguage model for unseen pill detection.…”

Section: B Zero-shot Object Detectionmentioning

confidence: 99%

Zero-Shot Pill-Prescription Matching With Graph Convolutional Network and Contrastive Learning

Nguyen,

Le Nguyen,

Kawanishi

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Patients' safety is paramount in the healthcare industry, and reducing medication errors is essential for improvement. A promising solution to this problem involves the development of automated systems capable of assisting patients in verifying their pill intake mistakes. This paper investigates a Pill-Prescription matching task that seeks to associate pills in a multi-pill photo with their corresponding names in the prescription. We specifically aimed to overcome the limitations of existing pill detection methods when faced with unseen pills, a situation characteristic of zero-shot learning. We propose a novel method named Zero-PIMA (Zero-shot Pill-Prescription Matching), designed to match pill images with prescription names effectively, even for pills not included in the training dataset. The Zero-PIMA is an end-to-end model that includes an object localization module to determine and extract features of pill images and a graph convolutional network to capture the spatial relationship of the pills' text in the prescription. After that, we leverage the contrastive learning paradigm to increase the distance between mismatched pill images and pill name pairs while minimizing the distance between matched pairs. In addition, to deal with the zero-shot pill detection problem, we leverage pills' metadata retrieved from the DrugBank database to fine-tune a pre-trained text encoder, thereby incorporating visual information about pills (e.g., shape, color) into their names, making them more informative and ultimately enhancing the pill image-name matching accuracy. Extensive experiments are conducted on our collected real-world VAIPE PP dataset of multi-pill photos and prescriptions. Through a series of comprehensive experiments, the proposed method outperforms other methods for both seen and unseen pills in terms of mean average precision. These results indicate that the proposed method could reduce medication errors and improve patients' safety.

show abstract

“…This setting hampers the broader applicability of SGG models in diverse real-world applications. Influenced by the achievements in open vocabulary object detection [7,14,36,41,47], recent works [9,45] attempt to extend the SGG task from closed-set to open vocabulary domain. However, they focus on an object-centric open vocabulary setting, which only considers the scene graph nodes.…”

Section: Introductionmentioning

confidence: 99%