Relation Distillation Networks for Video Object Detection

Abstract

It has been well recognized that modeling object-toobject relations would be helpful for object detection. Nevertheless, the problem is not trivial especially when exploring the interactions between objects to boost video object detectors. The difficulty originates from the aspect that reliable object relations in a video should depend on not only the objects in the present frame but also all the supportive objects extracted over a long range span of the video. In this paper, we introduce a new design to capture the interactions across the objects in spatio-temporal context. Specifically, we present Relation Distillation Networks (RDN) -a new architecture that novelly aggregates and propagates object relation to augment object features for detection. Technically, object proposals are first generated via Region Proposal Networks (RPN). RDN then, on one hand, models object relation via multi-stage reasoning, and on the other, progressively distills relation through refining supportive object proposals with high objectness scores in a cascaded manner. The learnt relation verifies the efficacy on both improving object detection in each frame and box linking across frames. Extensive experiments are conducted on Im-ageNet VID dataset, and superior results are reported when comparing to state-of-the-art methods. More remarkably, our RDN achieves 81.8% and 83.2% mAP with ResNet-101 and ResNeXt-101, respectively. When further equipped with linking and rescoring, we obtain to-date the best reported mAP of 83.8% and 84.7%.

Video Object Detection With Two-Path Convolutional LSTM Pyramid

“…Causal? Backbone mAP(%) mAP gain(%) T-CNN [13] No GoogLeNet + VGG + Fast-RCNN 73.8 6.1 MANet [14] No ResNet101 + R-FCN 78.1 4.5 FGFA [16] No ResNet101 + R-FCN 78.4 5.0 Scale-time lattice [20] No ResNet101+ Faster R-CNN 79.6 N/A Object linking [30] No ResNet101+ Fast R-CNN 74.5 5.4 Seq-NMS [19] No VGG + Faster R-CNN 52.2 7.3 STMN [18] No ResNet101 + R-FCN 80.5 N/A STSN [21] No ResNet101 + R-FCN 78.9 2.9 RDN [41] No ResNet101 + Faster R-CNN 81.8 6.4 SELSA [42] No ResNet101 + Faster R-CNN 80.3 6.7 D&T [15] No mance despite the fact that a less powerful detection network is used. Since our method focuses on causal video object detection where no future frames are allowed, no video-level post-processing is applied.…”

“…Cuboid proposal network and tubelet linking algorithm are proposed in [30] to improve the performance of detecting moving objects in videos. In [41], objects' interactions are captured in spatio-temporal domain. Full-sequence level feature aggregation is proposed in [42] to generate robust features for video object detection.…”

Single Shot Video Object Detector

“…STMN [22] adopts spatiotemporal memory module with spatial alignment mechanism to model long-term temporal appearance and motion dynamics. Besides, RDN [46] and SELSA [47] strengthen region-level features by exploiting the relation/affinity between region proposals across frames…”

Core-Text: Improving Scene Text Detection with Contrastive Relational Reasoning

“…Relational Reasoning. There has been strong evidences on the use of relational reasoning to support various tasks, e.g., object detection [11,14,15,16], feature learning [17], vision-language [18,19]. For example, [16] plugs non-local operation into the conventional CNN to enable the pixel-level relational interaction within feature maps, and [11] presents an object relation module to model the relations of regions via the interaction among appearance features and geometry.…”