OvarNet: Towards Open-vocabulary Object Attribute Recognition

Abstract

In this paper, we consider the problem of simultaneously detecting objects and inferring their visual attributes in an image, even for those with no manual annotations provided at the training stage, resembling an open-vocabulary scenario. To achieve this goal, we make the following contributions: (i) we start with a naïve two-stage approach for open-vocabulary object detection and attribute classification, termed CLIP-Attr. The candidate objects are first proposed with an offline RPN and later classified for semantic category and attributes; (ii) we combine all available datasets and train with a federated strategy to finetune the CLIP model, aligning the visual representation with attributes, additionally, we investigate the efficacy of leveraging freely available online image-caption pairs under weakly supervised learning; (iii) in pursuit of efficiency, we train a Faster-RCNN type model end-to-end with knowledge distillation, that performs class-agnostic object proposals and classification on semantic categories and attributes with classifiers generated from a text encoder; Finally, (iv) we conduct extensive experiments on VAW, MS-COCO, LSA, and OVAD datasets, and show that recog-Equal contribution. † Corresponding author. nition of semantic category and attributes is complementary for visual scene understanding, i.e., jointly training object detection and attributes prediction largely outperform existing approaches that treat the two tasks independently, demonstrating strong generalization ability to novel attributes and categories.

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation

“…Several studies have been conducted consecutively on learning multimodal structural knowledge, commencing with visual concept recognition as a basic task (Deng et al, 2009;Davoodi et al, 2023;Kamath et al, 2022), progressing to object grounding (Yu and Ballard, 2004;Shao et al, 2019;Krasin et al, 2017), object attribute detection (Chen et al, 2023a;Patil and Abhyankar, 2023;Bravo et al, 2022), object-object relation detection (Krishna et al, 2016), and finally visual event understanding (Yatskar et al, 2016(Yatskar et al, , 2017Cho et al, 2022;…”