Disentangling Visual Embeddings for Attributes and Objects

Saini, Nirat; Pham, Khoi; Shrivastava, Abhinav

doi:10.1109/cvpr52688.2022.01329

Cited by 46 publications

(18 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluate the two sub-tasks in both box-given and boxfree settings on COCO for category detection and VAW, LSA, and OVAD for attribute prediction. Specifically, the box-given setting is widely used in attribute prediction and object recognition communities [8,29,38,40], where the ground-truth bounding box annotations are assumed to be available for all objects, and the protocol only evaluates object category classification and multi-label attribute classification with mAP metric; In contrast, the box-free setting favors a more challenging problem, as the model is also required to simultaneously localise the objects, and classify the semantic category and attributes.…”

Section: Evaluation Protocol and Metricsmentioning

confidence: 99%

“…In the literature, numerous work has shown that understanding the objects' attributes can greatly facilitate object recognition and detection, even with few or no examples of visual objects [6,18,25,43,53], for example, Farhadi et al proposed to shift the goal of object recognition from 'naming' to 'description', which allows naming familiar objects with attributes, but also to say something about unfamiliar objects ("hairy and four-legged", not just "unknown") [6]; Lampert et al considered the open-set object recognition, that aims to recognise objects by humanspecified high-level description, e.g., arbitrary semantic attributes, like shape, color, or even geographic information, instead of training images [18]. However, the problem considered in these seminal work tends to be a simplification from today's standard, for example, attribute classification are often trained and evaluated on object-centric images under the close-set scenario, i.e., assuming the bounding boxes/segmentation masks are given [13,29,38], or sometimes even the object category are known as a prior [26,29].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

OvarNet: Towards Open-vocabulary Object Attribute Recognition

Chen¹,

Jiang²,

Hu³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we consider the problem of simultaneously detecting objects and inferring their visual attributes in an image, even for those with no manual annotations provided at the training stage, resembling an open-vocabulary scenario. To achieve this goal, we make the following contributions: (i) we start with a naïve two-stage approach for open-vocabulary object detection and attribute classification, termed CLIP-Attr. The candidate objects are first proposed with an offline RPN and later classified for semantic category and attributes; (ii) we combine all available datasets and train with a federated strategy to finetune the CLIP model, aligning the visual representation with attributes, additionally, we investigate the efficacy of leveraging freely available online image-caption pairs under weakly supervised learning; (iii) in pursuit of efficiency, we train a Faster-RCNN type model end-to-end with knowledge distillation, that performs class-agnostic object proposals and classification on semantic categories and attributes with classifiers generated from a text encoder; Finally, (iv) we conduct extensive experiments on VAW, MS-COCO, LSA, and OVAD datasets, and show that recog-Equal contribution. † Corresponding author. nition of semantic category and attributes is complementary for visual scene understanding, i.e., jointly training object detection and attributes prediction largely outperform existing approaches that treat the two tasks independently, demonstrating strong generalization ability to novel attributes and categories.

show abstract

Section: Evaluation Protocol and Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

OvarNet: Towards Open-vocabulary Object Attribute Recognition

Chen¹,

Jiang²,

Hu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Li et al [15] disentangle attributes and objects with reversed attention. The second strategy directly predicts compositions by aligning images and textual labels in a shared space and searching for most similar compositions [13,26,35,42]. For example, Nagarajan et al [26] build a composition space by simulating all the visual changes of attributes performed on objects.…”

Section: Related Workmentioning

confidence: 99%

Diversity-Boosted Generalization-Specialization Balancing for Zero-Shot Learning

Liu

Chang

et al. 2023

IEEE Trans. Multimedia

View full text Add to dashboard Cite

The objective of Active Learning is to strategically label a subset of the dataset to maximize performance within a predetermined labeling budget. In this study, we harness features acquired through self-supervised learning. We introduce a straightforward yet potent metric, Cluster Distance Difference, to identify diverse data. Subsequently, we introduce a novel framework, Balancing Active Learning (BAL), which constructs adaptive sub-pools to balance diverse and uncertain data. Our approach outperforms all established active learning methods on widely recognized benchmarks by 1.20%. Moreover, we assess the efficacy of our proposed framework under extended settings, encompassing both larger and smaller labeling budgets. Experimental results demonstrate that, when labeling 80% of the samples, the performance of the current SOTA method declines by 0.74%, whereas our proposed BAL achieves performance comparable to the full dataset. Codes are available at https://github.com/JulietLJY/BAL.

show abstract

“…In compositional zero-shot learning, Yang et al [25] disentangle images into states and objects based on the causal effects in compositions. Saini et al [28] apply a state/object affinity network to disentangle the same states or objects from image pairs contrastively. Zhang et al [27] reconsider CZSL as an out-of-distribution generalization problem and use domain alignment in the gradient level to disentangle images into object-invariant and attribute-invariant features.…”

Section: Knowledge Disentanglement In Zero-shot Learningmentioning

confidence: 99%

“…Simple primitives in images are entangled visually [25]. Most current works [16], [19], [25], [26], [27], [28] view states or objects as the bases. They simulate the visual changes on bases caused by contextuality, inferring the possible entangled embedding.…”

Section: Introductionmentioning

confidence: 99%

Simple Primitives with Feasibility- and Contextuality-Dependence for Open-World Compositional Zero-shot Learning

Liu¹,

Li²,

Yao³

et al. 2022

Preprint

View full text Add to dashboard Cite

The task of Compositional Zero-Shot Learning (CZSL) is to recognize images of novel state-object compositions that are absent during the training stage. Previous methods of learning compositional embedding have shown effectiveness in closed-world CZSL. However, in Open-World CZSL (OW-CZSL), their performance tends to degrade significantly due to the large cardinality of possible compositions. Some recent works separately predict simple primitives (i.e., states and objects) to reduce cardinality. However, they consider simple primitives as independent probability distributions, ignoring the heavy dependence between states, objects, and compositions. In this paper, we model the dependence of compositions via feasibility and contextuality. Feasibility-dependence refers to the unequal feasibility relations between simple primitives, e.g., hairy is more feasible with cat than with building in the real world. Contextuality-dependence represents the contextual variance in images, e.g., cat shows diverse appearances under the state of dry and wet. We design Semantic Attention (SA) and generative Knowledge Disentanglement (KD) to learn the dependence of feasibility and contextuality, respectively. SA captures semantics in compositions to alleviate impossible predictions, driven by the visual similarity between simple primitives. KD disentangles images into unbiased feature representations, easing contextual bias in predictions. Moreover, we complement the current compositional probability model with feasibility and contextuality in a compatible format. Finally, we conduct comprehensive experiments to analyze and validate the superior or competitive performance of our model, Semantic Attention and knowledge Disentanglement guided Simple Primitives (SAD-SP), on three widely-used benchmark OW-CZSL datasets.

show abstract

Disentangling Visual Embeddings for Attributes and Objects

Cited by 46 publications

References 35 publications

OvarNet: Towards Open-vocabulary Object Attribute Recognition

OvarNet: Towards Open-vocabulary Object Attribute Recognition

Diversity-Boosted Generalization-Specialization Balancing for Zero-Shot Learning

Simple Primitives with Feasibility- and Contextuality-Dependence for Open-World Compositional Zero-shot Learning

Contact Info

Product

Resources

About