Dual Semantic Fusion Network for Video Object Detection

Abstract

Video object detection is a tough task due to the deteriorated quality of video sequences captured under complex environments. Currently, this area is dominated by a series of feature enhancement based methods, which distill beneficial semantic information from multiple frames and generate enhanced features through fusing the distilled information. However, the distillation and fusion operations are usually performed at either frame level or instance level with external guidance using additional information, such as optical flow and feature memory. In this work, we propose a dual semantic fusion network (abbreviated as DSFNet) to fully exploit both frame-level and instance-level semantics in a unified fusion framework without external guidance. Moreover, we introduce a geometric similarity measure into the fusion process to alleviate the influence of information distortion caused by noise. As a result, the proposed DSFNet can generate more robust features through the multi-granularity fusion and avoid being affected by the instability of external guidance. To evaluate the proposed DSFNet, we conduct extensive experiments on the ImageNet VID dataset. Notably, the proposed dual semantic fusion network achieves, to the best of our knowledge, the best performance of 84.1% mAP among the current state-of-the-art video object detectors with ResNet-101 and 85.4% mAP with ResNeXt-101 without using any post-processing steps. CCS CONCEPTS• Computing methodologies → Object detection.

TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers

“…For the first aspect, most previous works [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [27], [28], [29] to amend this problem is feature aggregation that enhances per-frame features by aggregating the features of nearby frames. Earlier works adopt flow-based warping to achieve feature aggregation.…”

“…Faster RCNN 80.6 LRTR [35] Faster RCNN 81.0 RDN [34] Faster RCNN 81.8 TROI [72] Faster RCNN 82.0 MEGA [23] Faster RCNN 82.9 HVRNet [18] Faster RCNN 83.2 TF-Blender [73] Faster RCNN 83.8 DSFNet [20] Faster RCNN 84.1 MAMBA [24] Faster RCNN 84.6 EBFA [19] Faster RCNN 84.8 CFA-Net [74] Faster…”

“…These methods usually apply a still-image detector to obtain detection results, then associate the results. Another line of approaches [16], [17], [18], [19], [20], [21], [22], [23], [24] exploits the feature aggregation of temporal information. Specifically, they mainly improve features of the current frame by aggregating that of adjacent frames or entire clips to boost the detection performance via specific operator design.…”

Crossover Learning for Fast Online Video Instance Segmentation

“…However, extending these methods to VIS remains a challenging work. Similar to other video-based recognition tasks, such as video object segmentation (VOS) [49,50,33], video object detection (VOD) [53,37] and multi-object tracking (MOT) [16,21,59,75], continuous video sequences always bring great challenges, e.g., a huge number of frames required to be fast recognized, heavy occlusion, object disappearing and unconventional object-to-camera poses [18].…”

Tracking Instances as Queries

“…This task extends the traditional instance segmentation to the temporal domain and requires detecting, classifying, segmenting, and tracking visual instances simultaneously in the given videos. Similar to other video based tasks like Video Object Segmentation [12,25] and Video Object Detection [14], video instance segmentation provides a natural understanding of video scenes. Achieving accurate and robust video instance segmentation in realworld scenarios can greatly promote the development of video analysis.…”