GTNet: Generative Transfer Network for Zero-Shot Object Detection

Abstract

We propose a Generative Transfer Network (GTNet) for zero-shot object detection (ZSD). GTNet consists of an Object Detection Module and a Knowledge Transfer Module. The Object Detection Module can learn large-scale seen domain knowledge. The Knowledge Transfer Module leverages a feature synthesizer to generate unseen class features, which are applied to train a new classification layer for the Object Detection Module. In order to synthesize features for each unseen class with both the intra-class variance and the IoU variance, we design an IoU-Aware Generative Adversarial Network (IoUGAN) as the feature synthesizer, which can be easily integrated into GTNet. Specifically, IoUGAN consists of three unit models: Class Feature Generating Unit (CFU), Foreground Feature Generating Unit (FFU), and Background Feature Generating Unit (BFU). CFU generates unseen features with the intra-class variance conditioned on the class semantic embeddings. FFU and BFU add the IoU variance to the results of CFU, yielding class-specific foreground and background features, respectively. We evaluate our method on three public datasets and the results demonstrate that our method performs favorably against the state-of-the-art ZSD approaches.

Robust Region Feature Synthesizer for Zero-Shot Object Detection

“…Zero-shot object detection. ZSD receives great research interest in recent years [4,7,16,22,31,32,[41][42][43]. Some researches focus on embedding function-based methods [4,7,22,31,32,42].…”

“…Although some existing approaches have recognized the importance of intra-class diversity [16,41], without jointly considering the inter-class separability, these methods would either impose insufficient diversity to the synthesized visual features, leading to miss-classify the real unseen objects as image backgrounds (see Fig To overcome the feature synthesizing problems toward real-world detection scenarios, we build a novel zero-shot object detection framework as shown in Fig 2 . Specifically, we design two components for learning robust region features. To enable the model to synthesize diverse visual features, we propose an Intra-class Semantic Diverging (In-traSD) component which can diverge the semantic vector of a single class into a set of visual features.…”

Zero-Shot Object Detection With Transformers

“…In order to prove that our transformer encoder and decoder is more effective than the previous methods in recalling unseen objects, we train a RPN (the region proposal network) for comparison. We choose RPN here because that previous methods [7,10,12] have used it as unseen objects extractor. We use the same dataset and same backbone network with our ZSDTR to train the RPN and add the FPN [18] in it for fair comparison.…”

“…To remedy this problem, Bansal et al [6] and Rahman et al [7] first propose the definition of ZSD task, which aims to recognize and localize the unseen objects simultaneously. After that, several methods have been proposed to improve the performance for ZSD [8,9,10,11,12,13]. These methods use additional information like the attribute or the semantic word-vectors as extra knowledge to detect unseen objects, and we also use the semantic word-vectors in our method.…”

“…Our ZSDTR is an end-to-end zero-shot detector that is composed of the head network, a transformer-based encoder-decoder, and a vision-semantic-attention tail. Compared with the previous zero-shot object detectors [9,10,11,12,13], our ZS-DTR has a stronger ability to recall unseen objects and does not need to design a special method to deal with the confusion between unseen objects and background. We decompose the ZSD task into a class-agnostic regression problem and a zeroshot classification problem through the tail network, which learns the mapping relationship between vision objects and semantic word-vector from seen classes data and transfer it to detect unseen objects.…”