Zero-Shot Object Detection with Textual Descriptions

Abstract

Object detection is important in real-world applications. Existing methods mainly focus on object detection with sufficient labelled training data or zero-shot object detection with only concept names. In this paper, we address the challenging problem of zero-shot object detection with natural language description, which aims to simultaneously detect and recognize novel concept instances with textual descriptions. We propose a novel deep learning framework to jointly learn visual units, visual-unit attention and word-level attention, which are combined to achieve word-proposal affinity by an element-wise multiplication. To the best of our knowledge, this is the first work on zero-shot object detection with textual descriptions. Since there is no directly related work in the literature, we investigate plausible solutions based on existing zero-shot object detection for a fair comparison. We conduct extensive experiments on three challenging benchmark datasets. The extensive experimental results confirm the superiority of the proposed model.

GTNet: Generative Transfer Network for Zero-Shot Object Detection

“…The study (Li et al 2019b) extends other the other two ZSD methods Bansal et al 2018) using the textual description embedding and report the results. We just copy and paste the results from the paper (Li et al 2019b) in Table 1 and Table 2.…”

“…All the modules are trained using the Adam optimizer with a 10 −4 learning rate. Compared Methods: We compare the effect of our proposed approach to three recent state-of-the-art methods Bansal et al 2018;Li et al 2019b). In order to fairly compare with the method (Li et al 2019b), we use the textual description embedding which is trained by fastText (Edouard Grave and Bojanowski 2017) as the class semantic embedding.…”

“…Compared Methods: We compare the effect of our proposed approach to three recent state-of-the-art methods Bansal et al 2018;Li et al 2019b). In order to fairly compare with the method (Li et al 2019b), we use the textual description embedding which is trained by fastText (Edouard Grave and Bojanowski 2017) as the class semantic embedding. The study (Li et al 2019b) extends other the other two ZSD methods Bansal et al 2018) using the textual description embedding and report the results.…”

“…In order to fairly compare with the method (Li et al 2019b), we use the textual description embedding which is trained by fastText (Edouard Grave and Bojanowski 2017) as the class semantic embedding. The study (Li et al 2019b) extends other the other two ZSD methods Bansal et al 2018) using the textual description embedding and report the results. We just copy and paste the results from the paper (Li et al 2019b) in Table 1 and Table 2.…”

“…(1) Existing methods (Bansal et al 2018;Li et al 2019b) generally map the visual features from predicted bounding boxes to the semantic embedding space. During the inference stage, a class label is predicted by finding the nearest class based on the similarity scores with all Figure 1: Illustration of our feature generating process.…”