Learning to discover and localize visual objects with open vocabulary

Ye, Keren; Zhang, Mingda; Li, Wei; Qin, Danfeng; Kovashka, Adriana; Berent, Jesse

doi:10.48550/arxiv.1811.10080

Cited by 3 publications

(3 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is similar to a closed vocabulary, yet the rich semantic content of captions is discarded. In contrast, research in [123] and [158] aims to discover an open set of object classes from image-caption corpora, and learns detectors for each discovered class.…”

Section: B) Weak Supervisionmentioning

confidence: 99%

Bridging Gap between Image Pixels and Semantics via Supervision: A Survey

Duan¹,

Kuo²

2021

Preprint

View full text Add to dashboard Cite

The fact that there exists a gap between low-level features and semantic meanings of images, called the semantic gap, is known for decades. Resolution of the semantic gap is a long standing problem. The semantic gap problem is reviewed and a survey on recent efforts in bridging the gap is made in this work. Most importantly, we claim that the semantic gap is primarily bridged through supervised learning today. Experiences are drawn from two application domains to illustrate this point: 1) object detection and 2) metric learning for content-based image retrieval (CBIR). To begin with, this paper offers a historical retrospective on supervision, makes a gradual transition to the modern data-driven methodology and introduces commonly used datasets. Then, it summarizes various supervision methods to bridge the semantic gap in the context of object detection and metric learning.

show abstract

Section: B) Weak Supervisionmentioning

confidence: 99%

Bridging Gap between Image Pixels and Semantics via Supervision: A Survey

Duan¹,

Kuo²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Amrani et al [2] train a WSD model based on the presence of a predefined set of words in captions, which is similarly closed-vocabulary, and discards the rich semantic content of captions, which we exploit through transformers. In contrast, Sun et al [35] and Ye et al [44] aim to discover an open set of object classes from image-caption corpora, and learn detectors for each discovered class. A key limitation of all such WSD methods is their inferior object localization accuracy.…”

Section: Related Workmentioning

confidence: 99%

Open-Vocabulary Object Detection Using Captions

Zareian¹,

Rosa²,

Hu³

et al. 2020

Preprint

View full text Add to dashboard Cite

Despite the remarkable accuracy of deep neural networks in object detection, they are costly to train and scale due to supervision requirements. Particularly, learning more object categories typically requires proportionally more bounding box annotations. Weakly supervised and zero-shot learning techniques have been explored to scale object detectors to more categories with less supervision, but they have not been as successful and widely adopted as supervised models. In this paper, we put forth a novel formulation of the object detection problem, namely openvocabulary object detection, which is more general, more practical, and more effective than weakly supervised and zero-shot approaches. We propose a new method to train object detectors using bounding box annotations for a limited set of object categories, as well as image-caption pairs that cover a larger variety of objects at a significantly lower cost. We show that the proposed method can detect and localize objects for which no bounding box annotation is provided during training, at a significantly higher accuracy than zero-shot approaches. Meanwhile, objects with bounding box annotation can be detected almost as accurately as supervised methods, which is significantly better than weakly supervised baselines. Accordingly, we establish a new state of the art for scalable object detection.

show abstract

“…[27] extracts names and actions from captions and associates them to the faces and body poses of people shown in the images, to train face and pose models. In a similar spirit, [42] proposes a method to detect objects from captions. While these methods directly use the raw, noisy data, we propose a method to clean and homogenize annotations into a natural vocabulary.…”

Section: Related Workmentioning

confidence: 99%

Natural Vocabulary Emerges from Free-Form Annotations

Pont-Tuset¹,

Gygli²,

Ferrari³

2019

Preprint

View full text Add to dashboard Cite

We propose an approach for annotating object classes using free-form text written by undirected and untrained annotators. Free-form labeling is natural for annotators, they intuitively provide very specific and exhaustive labels, and no training stage is necessary. We first collect 729k labels on 15k images using 124 different annotators. Then we automatically enrich the structure of these free-form annotations by discovering a natural vocabulary of 4,020 classes within them. This vocabulary represents the natural distribution of objects well and is learned directly from data, instead of being an educated guess done before collecting any labels. Hence, the natural vocabulary emerges from a large mass of free-form annotations. To do so, we (i) map the raw input strings to entities in an ontology of physical objects (which gives them an unambiguous meaning); and (ii) leverage inter-annotator co-occurrences, as well as biases and knowledge specific to individual annotators. Finally, we also automatically extract natural vocabularies of reduced size that have high object coverage while remaining specific. These reduced vocabularies represent the natural distribution of objects much better than commonly used predefined vocabularies. Moreover, they feature more uniform sample distribution over classes.

show abstract

Learning to discover and localize visual objects with open vocabulary

Cited by 3 publications

References 41 publications

Bridging Gap between Image Pixels and Semantics via Supervision: A Survey

Bridging Gap between Image Pixels and Semantics via Supervision: A Survey

Open-Vocabulary Object Detection Using Captions

Natural Vocabulary Emerges from Free-Form Annotations

Contact Info

Product

Resources

About