Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation

Jianzong, Wu,; Li, Xiangtai; Ding, Henghui; Loy, Chen Change

doi:10.36227/techrxiv.22082723.v1

Cited by 8 publications

(5 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(1), The task definitions and settings are different. In addition to different input and output formats, taking the open vocabulary segmentation as an example, there are at least three different settings [46], [47], [125] for open vocabulary segmentation. Thus, the design principles are different.…”

Section: Methods: a Surveymentioning

confidence: 99%

“…(2), Different methods use different detectors or baseline models. Several works [11], [134] use R-CNN framework while several methods [47], [131] adopt query-based approaches. Several methods [136], [137] adopt the RPN's features to develop their methods, while query-based methods cannot perform these designs.…”

Section: Methods: a Surveymentioning

confidence: 99%

“…• Tasks. Open vocabulary learning has included a wide range of computer vision tasks, including object detection [11], [44], [45], segmentation [46], [47], [48], video understanding, and 3D scene understanding. The center goal of these tasks is similar, recognizing the novel classes with the aid of large vocabulary knowledge for their corresponding tasks.…”

Section: Tasks Datasets and Metricsmentioning

confidence: 99%

“…ViLD [12] BARON [113] LSeg [14] ZegFormer [114] MaskCLIP [48] OPSNet [115] OpenScene [116] ULIP [117] OVR-CNN [11] OpenSeg [46] CGG [47] DST-Det [118] PB-OVD [119] MaskCLIP+ [120] OVSeg [121] XPM [122] MAXI [123] PLA [124] GroupViT [125] SegCLIP [126] Mask-free OVIS [127] OpenSeeD [128] OpenSD [129] X-Decoder [130] FreeSeg [131] OVDiff [132] ODISE [133] Detic [134] OWLv2 [135] Fig. 4: Open vocabulary learning methods, organized by their tasks and approach types.…”

Section: Preliminarymentioning

confidence: 99%

“…OpenSeg [46] applies a region-word grounding loss to directly ground objects and nouns in the caption data. CGG [47] combines caption grounding and caption generation losses to fully exploit the knowledge in caption data. It leverages the role of object nouns in visual grounding and the mutual benefits of words in caption generation.…”

Section: Open Vocabulary Segmentationmentioning

confidence: 99%

See 4 more Smart Citations

Towards Open Vocabulary Learning: A Survey

Wu,

Li,

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

In the field of visual scene understanding, deep neural networks have made impressive advancements in various core tasks like segmentation, tracking, and detection. However, most approaches operate on the close-set assumption, meaning that the model can only identify pre-defined categories that are present in the training set. Recently, open vocabulary settings were proposed due to the rapid progress of vision language pre-training. These new approaches seek to locate and recognize categories beyond the annotated label space. The open vocabulary approach is more general, practical, and effective than weakly supervised and zero-shot settings. This paper thoroughly reviews open vocabulary learning, summarizing and analyzing recent developments in the field. In particular, we begin by juxtaposing open vocabulary learning with analogous concepts such as zero-shot learning, open-set recognition, and out-of-distribution detection. Subsequently, we examine several pertinent tasks within the realms of segmentation and detection, encompassing long-tail problems, few-shot, and zero-shot settings. As a foundation for our method survey, we first elucidate the fundamental principles of detection and segmentation in close-set scenarios. Next, we examine various contexts where open vocabulary learning is employed, pinpointing recurring design elements and central themes. This is followed by a comparative analysis of recent detection and segmentation methodologies in commonly used datasets and benchmarks. Our review culminates with a synthesis of insights, challenges, and discourse on prospective research trajectories. To our knowledge, this constitutes the inaugural exhaustive literature review on open vocabulary learning. We keep tracing related works at https://github.com/jianzongwu/Awesome-Open-Vocabulary.

show abstract

Section: Methods: a Surveymentioning

confidence: 99%