Polarity Loss for Zero-shot Object Detection

Abstract

Zero-shot object detection is an emerging research topic that aims to recognize and localize previously 'unseen' objects. This setting gives rise to several unique challenges, e.g., highly imbalanced positive vs. negative instance ratio, ambiguity between background and unseen classes and the proper alignment between visual and semantic concepts. Here, we propose an end-to-end deep learning framework underpinned by a novel loss function that seeks to properly align the visual and semantic cues for improved zero-shot learning. We call our objective the 'Polarity loss' because it explicitly maximizes the gap between positive and negative predictions. Such a margin maximizing formulation is not only important for visual-semantic alignment but it also resolves the ambiguity between background and unseen objects. Our approach is inspired by the embodiment theories in cognitive science, that claim human semantic understanding to be grounded in past experiences (seen objects), related linguistic concepts (word dictionary) and the perception of the physical world (visual imagery). To this end, we learn to attend to a dictionary of related semantic concepts that eventually refines the noisy semantic embeddings and helps establish a better synergy between visual and semantic domains. Our extensive results on MS-COCO and Pascal VOC datasets show as high as 14× mAP improvement over state of the art. 1

Caption Generation on Scenes with Seen and Unseen Object Categories

“…Region proposal-based approaches instead first generate region proposals, then classify (and update) each region proposal [3], [4], [6], [7], [10], [55], [56]. Recently, zero-shot object detection methods have been proposed [23], [24], [25], [26], [27], [57], [58], typically as extensions of supervised object detection models. Among these studies, Bansal et al [25] proposes a two-step approach which first locates object proposals from low-level features [59] and then classifies the resulting candidate regions using a ZSL model.…”

“…Demirel et al [23] proposes a regressionbased ZSD model that jointly incorporates two ZSL models based on convex combinations of semantic embeddings [60] and bi-linear compatibility models [16]. Rahman et al [26] proposes a polarity loss term that is based on the focal loss approach, to tackle better alignment between visual and semantic domains. Hence, semantic representations of visually similar classes get closer to each other.…”

“…This work was supported in part by the TUBITAK Grant 116E445 and 119E597. methods have also been proposed for the zero-shot object detection (ZSD) problem [23], [24], [25], [26], [27].…”

Semantics-Guided Contrastive Network for Zero-Shot Object detection

“…In terms of the MS COCO dataset, we follow the same procedures described in [11] to divide the dataset into two different splits: (1) 48 seen and 17 unseen classes; (2) 65 seen and 15 unseen classes. Conditioned on the above seen/unseen class splits, we follow the steps in [62] to create the train and test set for each dataset. Implementation Details.…”

“…TD [14] learns both visual-unit-level and word-level attention to tackle the ZSD task with textual descriptions instead of a single word. PL [62] designs a novel polarity loss for RetinaNet based ZSD framework to better align visual and semantic concepts. BLC [56] combines Cascade Semantic R-CNN, semantic information flow and background learnable RPN into a unified framework for the ZSD task.…”