Deep Multiple Instance Learning for Zero-Shot Image Tagging

Rahman, Shafin; Khan, Salman

doi:10.1007/978-3-030-20887-5_33

Cited by 9 publications

(7 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ZSL literature is predominated by classification approaches that focus on single (Fu et al 2018) or multilabel (Rahman and Khan 2018) recognition. The ZSD problem has recently been investigated by (Rahman, Khan, and Porikli 2018b;Zhu et al 2018;Bansal et al 2018;Demirel, Cinbis, and Ikizler-Cinbis 2018;Li et al 2019).…”

Section: Related Workmentioning

confidence: 99%

Improved Visual-Semantic Alignment for Zero-Shot Object Detection

Rahman

Khan

Barnes

2020

AAAI

Self Cite

107

View full text Add to dashboard Cite

Zero-shot object detection is an emerging research topic that aims to recognize and localize previously ‘unseen’ objects. This setting gives rise to several unique challenges, e.g., highly imbalanced positive vs. negative instance ratio, proper alignment between visual and semantic concepts and the ambiguity between background and unseen classes. Here, we propose an end-to-end deep learning framework underpinned by a novel loss function that handles class-imbalance and seeks to properly align the visual and semantic cues for improved zero-shot learning. We call our objective the ‘Polarity loss’ because it explicitly maximizes the gap between positive and negative predictions. Such a margin maximizing formulation is not only important for visual-semantic alignment but it also resolves the ambiguity between background and unseen objects. Further, the semantic representations of objects are noisy, thus complicating the alignment between visual and semantic domains. To this end, we perform metric learning using a ‘Semantic vocabulary’ of related concepts that refines the noisy semantic embeddings and establishes a better synergy between visual and semantic domains. Our approach is inspired by the embodiment theories in cognitive science, that claim human semantic understanding to be grounded in past experiences (seen objects), related linguistic concepts (word vocabulary) and the visual perception (seen/unseen object images). Our extensive results on MS-COCO and Pascal VOC datasets show significant improvements over state of the art.1

show abstract

Section: Related Workmentioning

confidence: 99%

Improved Visual-Semantic Alignment for Zero-Shot Object Detection

Rahman

Khan

Barnes

2020

AAAI

Self Cite

107

View full text Add to dashboard Cite

show abstract

“…In this paper, we perform both transductive and inductive ZSL and GZSL on 3D point cloud objects. Zero-Shot Learning: For the ZSL task, there has been significant progress, including on image recognition [43,74,1,3,37,25,65], multi-label ZSL [26,42], and zero-shot detection [44]. Despite this progress, these methods solve the constrained problem where the test instances are restricted to only unseen classes, rather than being from either seen or unseen classes.…”

Section: Related Workmentioning

confidence: 99%

Zero-shot Learning of 3D Point Cloud Objects

Cheraghian

Rahman

2019

2019 16th International Conference on Machine Vision Applications (MVA)

Self Cite

View full text Add to dashboard Cite

Recent deep learning architectures can recognize instances of 3D point cloud objects of previously seen classes quite well. At the same time, current 3D depth camera technology allows generating/segmenting a large amount of 3D point cloud objects from an arbitrary scene, for which there is no previously seen training data. A challenge for a 3D point cloud recognition system is, then, to classify objects from new, unseen, classes. This issue can be resolved by adopting a zero-shot learning (ZSL) approach for 3D data, similar to the 2D image version of the same problem. ZSL attempts to classify unseen objects by comparing semantic information (attribute/word vector) of seen and unseen classes. Here, we adapt several recent 3D point cloud recognition systems to the ZSL setting with some changes to their architectures. To the best of our knowledge, this is the first attempt to classify unseen 3D point cloud objects in the ZSL setting. A standard protocol (which includes the choice of datasets and the seen/unseen split) to evaluate such systems is also proposed. Baseline performances are reported using the new protocol on the investigated models. This investigation throws a new challenge to the 3D point cloud recognition community that may instigate numerous future works.

show abstract

“…The ZSL literature is predominated by classification approaches that focus on single [12,45,21,35,49,19] or multi-label [22,34] recognition. The extension of conventional ZSL approaches to zero-shot object localization/detection is relatively less investigated.…”

Section: Related Workmentioning

confidence: 99%

“…[14,25] proposed novel object localization based on natural language description. Few other methods located unseen objects with weak image-level la-bels [34,39]. However, none of them perform ZSD.…”

Section: Related Workmentioning

confidence: 99%

Polarity Loss for Zero-shot Object Detection

Rahman,

Khan,

Barnes

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Zero-shot object detection is an emerging research topic that aims to recognize and localize previously 'unseen' objects. This setting gives rise to several unique challenges, e.g., highly imbalanced positive vs. negative instance ratio, ambiguity between background and unseen classes and the proper alignment between visual and semantic concepts. Here, we propose an end-to-end deep learning framework underpinned by a novel loss function that seeks to properly align the visual and semantic cues for improved zero-shot learning. We call our objective the 'Polarity loss' because it explicitly maximizes the gap between positive and negative predictions. Such a margin maximizing formulation is not only important for visual-semantic alignment but it also resolves the ambiguity between background and unseen objects. Our approach is inspired by the embodiment theories in cognitive science, that claim human semantic understanding to be grounded in past experiences (seen objects), related linguistic concepts (word dictionary) and the perception of the physical world (visual imagery). To this end, we learn to attend to a dictionary of related semantic concepts that eventually refines the noisy semantic embeddings and helps establish a better synergy between visual and semantic domains. Our extensive results on MS-COCO and Pascal VOC datasets show as high as 14× mAP improvement over state of the art. 1

show abstract

Deep Multiple Instance Learning for Zero-Shot Image Tagging

Cited by 9 publications

References 41 publications

Improved Visual-Semantic Alignment for Zero-Shot Object Detection

Improved Visual-Semantic Alignment for Zero-Shot Object Detection

Zero-shot Learning of 3D Point Cloud Objects

Polarity Loss for Zero-shot Object Detection

Contact Info

Product

Resources

About