ScarfNet: Multi-scale Features with Deeply Fused and Redistributed Semantics for Enhanced Object Detection

Abstract

Convolutional neural network (CNN) has led significant progress in object detection. In order to detect the objects in various sizes, the object detectors often exploit the hierarchy of the multi-scale feature maps called feature pyramid, which is readily obtained by the CNN architecture. However, the performance of these object detectors is limited since the bottom-level feature maps, which experience fewer convolutional layers, lack the semantic information needed to capture the characteristics of the small objects. In order to address such problem, various methods have been proposed to increase the depth for the bottom-level features used for object detection. While most approaches are based on the generation of additional features through the top-down pathway with lateral connections, our approach directly fuses multi-scale feature maps using bidirectional long short term memory (biLSTM) in effort to generate deeply fused semantics. Then, the resulting semantic information is redistributed to the individual pyramidal feature at each scale through the channel-wise attention model. We integrate our semantic combining and attentive redistribution feature network (ScarfNet) with the baseline object detectors, i.e., Faster R-CNN, single-shot multibox detector (SSD) and RetinaNet. Our experiments show that our method outperforms the existing feature pyramid methods as well as the baseline detectors and achieve the state of the art performances in the PASCAL VOC and COCO detection benchmarks.

Semantic Attention and Scale Complementary Network for Instance Segmentation in Remote Sensing Images

“…Libra-RCNN [39] fuses the 5-level feature maps from FPN [42] and uses a Gaussian Non-Local [43] attention to obtain the balance semantic features. ScarfNet [40] generates features with strong semantic attention for each pyramid scale by bidirectional long short term memory (biLSTM) [44] and channel-wise attention. DES [41], built upon SSD [45], adds an extra semantic attention branch supervised with weak segmentation ground-truth for semantic enrichment.…”

“…Yang et al [46] proposed a Multi-Dimensional Attention Network to strengthen the response of the region of interest. In contrast to [10], [37]- [40], our proposed method adds accurate supervision to guide the learning of the attention mechanism. Besides, our attention mechanism has a simple structure compared to the biLSTM in ScarfNet [40] and the atrous convolution in DES [41].…”