Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity

Abstract

DETR is the first end-to-end object detector using a transformer encoder-decoder architecture and demonstrates competitive performance but low computational efficiency on high resolution feature maps. The subsequent work, Deformable DETR, enhances the efficiency of DETR by replacing dense attention with deformable attention, which achieves 10× faster convergence and improved performance. Deformable DETR uses the multiscale feature to ameliorate performance, however, the number of encoder tokens increases by 20× compared to DETR, and the computation cost of the encoder attention remains a bottleneck. In our preliminary experiment, we observe that the detection performance hardly deteriorates even if only a part of the encoder token is updated. Inspired by this observation, we propose Sparse DETR that selectively updates only the tokens expected to be referenced by the decoder, thus help the model effectively detect objects. In addition, we show that applying an auxiliary detection loss on the selected tokens in the encoder improves the performance while minimizing computational overhead. We validate that Sparse DETR achieves better performance than Deformable DETR even with only 10% encoder tokens on the COCO dataset. Albeit only the encoder tokens are sparsified, the total computation cost decreases by 38% and the frames per second (FPS) increases by 42% compared to Deformable DETR. Code is available at https://github.com/kakaobrain/sparse-detr.

YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors

“…Its main concept is to add extra auxiliary head in the middle layers of the network, and the shallow network weights with assistant loss as the guide. Even for architectures such as ResNet [26] and DenseNet [32] which usually converge well, deep supervision [70,98,67,47,82,65,86,50] can still significantly improve the performance of the model on many tasks. Figure 5 (a) and (b) show, respectively, the object detector architecture "without" and "with" deep supervision.…”

Transformer-Based Approach for Document Layout Understanding

“…The novelty of this study is to combine merits of the most recent transformer-based works in CV, including the top-k object query [11], bounding box refinement and two-stage strategy [9], and auxiliary losses in encoder layer [12] to improve the performance in terms of accuracy and efficiency. This combination is integrated into the implementation of the transformer-based encoder-decoder detector which follows the structure in Deformable DETR.…”

FAQ: Feature Aggregated Queries for Transformer-based Video Object Detectors

“…With the development of deep learning, extraordinary processes have been achieved in static image object detection [3,18,38,43,63]. Existing object detectors can be mainly divided into three categories: twostage [3,25,28,42,57], one-stage [43, 45, 49, 54-56, 64, 65] and query-based models [4,23,48,58,63,90]. For better performance, two-stage models generate a set of proposals and then refine the prediction results, like R-CNN fami-lies [11,22,28,57].…”