Background Learnable Cascade for Zero-Shot Object Detection

Abstract

Zero-shot detection (ZSD) is crucial to large-scale object detection with the aim of simultaneously localizing and recognizing unseen objects. There remain several challenges for ZSD, including reducing the ambiguity between background and unseen objects as well as improving the alignment between visual and semantic concept. In this work, we propose a novel framework named Background Learnable Cascade (BLC) to improve ZSD performance. The major contributions for BLC are as follows: (i) we propose a multi-stage cascade structure named Cascade Semantic R-CNN to progressively refine the alignment between visual and semantic of ZSD; (ii) we develop the semantic information flow structure and directly add it between each stage in Cascade Semantic R-CNN to further improve the semantic feature learning; (iii) we propose the background learnable region proposal network (BLRPN) to learn an appropriate word vector for background class and use this learned vector in Cascade Semantic R-CNN, this design makes "Background Learnable" and reduces the confusion between background and unseen classes. Our extensive experiments show BLC obtains significantly performance improvements for MS-COCO over state-of-the-art methods.

Zero-Shot Instance Segmentation

“…We compare our method with the state-of-the-art zeroshot detection approaches on two split benchmarks in Table 2 over ZSD setting. We can observe that: (i) for 48/17 split, we compare our approaches with SB [23], DSES [23], TD [29], PL [26], Gtnet [30], DELO [31] and BLC [32]. Our method surpasses all of them, brings up to 36.99% and 11.08% gain in terms of Recall@100 and mAP; (ii) for 65/15 split, compared with PL [26] and BLC [32], our method still gets the best performance and brings 21.18% gain for Recall@100 and 1.2% improvement for mAP.…”

“…Zhu et al [31] propose DELO that synthesizes visual features for unseen objects from semantic information and incorporate the detection for seen and unseen objects. Zheng et al [32] boost the ZSD performance with a cascade structure to progressively refine the visual-semantic mapping relationship and propose BLRPN to learn the word-vector for background class. In terms of the background representation, SB [23] and PL [26] use the fixed word-vector for "background" word, DSES [23] and BLC [32] try to learn a more reasonable representation.…”

“…Zheng et al [32] boost the ZSD performance with a cascade structure to progressively refine the visual-semantic mapping relationship and propose BLRPN to learn the word-vector for background class. In terms of the background representation, SB [23] and PL [26] use the fixed word-vector for "background" word, DSES [23] and BLC [32] try to learn a more reasonable representation. However, the training methods for DSES [23] and BLC [32] are multi-step processes.…”

“…In terms of the background representation, SB [23] and PL [26] use the fixed word-vector for "background" word, DSES [23] and BLC [32] try to learn a more reasonable representation. However, the training methods for DSES [23] and BLC [32] are multi-step processes. The background wordvector learned from these non-end-to-end training processes is a local optimum result and it is used as a fixed representation.…”

“…GZSI is closer to the situation in real world because the seen and unseen instances may appear at the same time. Referring to the previous works of ZSD [23,26,32,30], we use Recall@100 across different IoU thresholds (0.4, 0.5, 0.6) as the main metric and the 100 means we select the top 100 score results for evaluation. In addition, we also give the results of mean average precision (IoU threshold is 0.5) as a reference.…”