Improving Video Object Detection by Seq-Bbox Matching

Belhassen, Hatem; Zhang, Heng; Fresse, Virginie; Bourennane, El-Bay

doi:10.5220/0007260002260233

Cited by 24 publications

(18 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the great success of these approaches, most of the pipelines for video object detection are too sophisticated, requiring many hand-crafted components, e.g., extra optic flow model, memory mechanism, or recurrent neural network. In addition, most of them need complicated post-processing methods such as Seq-NMS [12], Tubelet rescoring [13], Seq-Bbox Matching [14] or REPP [15] For the second aspect, starting from DFF [26], several works [16], [17], [33], [36], [37], [38], [52], [53] focus on realtime video object detection while keeping accuracy unchanged or even improved. In general, most of these works also perform specific architecture design with many hand-crafted components and human prior such as object-level tracker in [16], patchwork cell with attention in [53] and Convolutional LSTMs in [37].…”

Section: Related Workmentioning

confidence: 99%

“…Previous video object detection approaches mainly leverage temporal information in two different manners. The first one relies on post-processing of temporal information to make the object detection results more coherent and stable [12], [13], [14], [15]. These methods usually apply a still-image detector to obtain detection results, then associate the results.…”

Section: Introductionmentioning

confidence: 99%

“…Despite the gratifying success of these approaches, most of the two-stage pipelines for video object detection are over sophisticated, requiring many hand-crafted components, e.g., optical flow model [26], [27], [28], [29], [30], recurrent neural network [23], [25], [31], deformable convolution fusion [21], [32], [33], relation networks [23], [34], [35]. In addition, most of them need complicated post-processing methods by linking the same object across the video to form tubelets and aggregating classification scores in the tubelets to achieve the state-of-the-art performance [12], [13], [14], [15]. Meanwhile, there are also several related studies [16], [17], [33], [36], [37], [38] focusing on real-time video object detection.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers

Zhou,

Li,

et al. 2022

Preprint

View full text Add to dashboard Cite

Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, the first end-to-end video object detection system based on spatial-temporal Transformer architectures. The first goal of this paper is to streamline the pipeline of VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow model, relation networks. Besides, benefited from the object query design in DETR, our method does not need complicated post-processing methods such as Seq-NMS. In particular, we present a temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal transformer consists of two components: Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3 %-4 % mAP) on the ImageNet VID dataset. TransVOD yields comparable performances on the benchmark of ImageNet VID. Then, we present two improved versions of TransVOD including TransVOD++ and TransVOD Lite. The former fuses object-level information into object query via dynamic convolution while the latter models the entire video clips as the output to speed up the inference time. We give detailed analysis of all three models in the experiment part. In particular, our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0 % mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7 % mAP while running at around 30 FPS on a single V100 GPU device. Code and models will be available for further research.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers

Zhou,

Li,

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The first * The first three authors contribute equally to this work. one relies on post-processing of temporal information to make the object detection results more coherent and stable [1,19,27,31,41,47]. These methods usually apply a still-image detector to obtain detection results, then associate the results.…”

Section: Introductionmentioning

confidence: 99%

“…Despite the gratifying success of these approaches, most of the two-stage pipelines for video object detection are over sophisticated, requiring many hand-crafted components, e.g., optical flow model [50,58,60,61], recurrent neural network [5,9,16], deformable convolution fusion [2,20,25], relation networks [5,11,42]. In addition, most of them need complicated post-processing methods by linking the same object across the video to form tubelets and aggregating classification scores in the tubelets to achieve the state-of-the-art performance [1,19,27,41]. Thus, it is in desperate need to build a simple yet effective VOD framework in a fully end-to-end manner.…”

Section: Introductionmentioning

confidence: 99%

End-to-End Video Object Detection with Spatial-Temporal Transformers

Lű¹,

Zhou²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, DETR and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, an end-to-end video object detection model based on a spatial-temporal Transformer architecture. The goal of this paper is to streamline the pipeline of VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow, recurrent neural networks, relation networks. Besides, benefited from the object query design in DETR, our method does not need complicated post-processing methods such as Seq-NMS or Tubelet rescoring, which keeps the pipeline simple and clean. In particular, we present temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal Transformer consists of three components: Temporal Deformable Transformer Encoder (TDTE) to encode the multiple frame spatial details, Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3%-4% mAP) on the ImageNet VID dataset. TransVOD yields comparable results performance on the benchmark of ImageNet VID. We hope our TransVOD can provide a new perspective for video object detection. Code will be made publicly available at https://github.com/SJTU-LuHe/TransVOD.

show abstract