End-to-End Video Instance Segmentation with Transformers

Wang, Yuqing; Xu, Zhaoliang; Wang, Xinlong; Shen, Chunhua; Cheng, Baoshan; Shen, Hao; Xia, Huaxia

doi:10.48550/arxiv.2011.14503

Cited by 56 publications

(63 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Vision Transformers [10], [11], [40], [41], [47], [54] make a great progress the computer vision. It can be mainly divided into two directions: replacing CNN backbone with Transformer-Like architecture [40], [47], [55] and using object query to represent instance for scene understanding [10], [11], [39]. Our work is related to the second part.…”

Section: Related Workmentioning

confidence: 99%

“…We suppose that this way can not fully use enough temporal contexts from a video clip. VisTR [39] views the VIS task as a direct end-to-end parallel sequence prediction problem. The targets of a clip are disrupted in such an instance sequence and directly performing target assignment is not optimal.…”

Section: Related Workmentioning

confidence: 99%

“…The targets of a clip are disrupted in such an instance sequence and directly performing target assignment is not optimal. Instead, we aim to link the outputs of the spatial Transformer, i.e., object query, through a temporal Transformer, which acts in a completely different way from VisTR [39]. To our knowledge, there are no prior applications of Transformers to video object detection (VOD) tasks so far.…”

Section: Related Workmentioning

confidence: 99%

“…Transformers [10], [11], [39], [40], [41] have shown promising potential in computer vision. Especially, DETR [10], [11] simplifies the detection pipeline by modeling the object queries and achieving comparative performance with highly optimized CNNbased detectors.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers

Zhou,

Li,

et al. 2022

Preprint

View full text Add to dashboard Cite

Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, the first end-to-end video object detection system based on spatial-temporal Transformer architectures. The first goal of this paper is to streamline the pipeline of VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow model, relation networks. Besides, benefited from the object query design in DETR, our method does not need complicated post-processing methods such as Seq-NMS. In particular, we present a temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal transformer consists of two components: Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3 %-4 % mAP) on the ImageNet VID dataset. TransVOD yields comparable performances on the benchmark of ImageNet VID. Then, we present two improved versions of TransVOD including TransVOD++ and TransVOD Lite. The former fuses object-level information into object query via dynamic convolution while the latter models the entire video clips as the output to speed up the inference time. We give detailed analysis of all three models in the experiment part. In particular, our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0 % mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7 % mAP while running at around 30 FPS on a single V100 GPU device. Code and models will be available for further research.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers

Zhou,

Li,

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, the Vision Transformer (ViT) structure (Dosovitskiy et al, 2020) firstly introduces the transformers into the image classification task and suggests that a convolution-free architecture can achieve state-of-the-art performance. Later on, transformers have been widely used in several vision tasks performance, such as detection (Carion et al, 2020;Zhu et al, 2020), segmentation (Zheng et al, 2020;Wang et al, 2020), and pose estimation (Lin et al, 2020;Li et al, 2021). However, the excellent performance improvement requires increasing the model size and computation complexity, and it is difficult to deploy these huge models into real-world applications like augmented reality and autonomous driving.…”

Section: Introductionmentioning

confidence: 99%

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

Sun¹,

Ma²,

Kang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The transformer architectures with attention mechanisms have obtained success in Nature Language Processing (NLP), and Vision Transformers (ViTs) have recently extended the application domains to various vision tasks. While achieving high performance, ViTs suffer from large model size and high computation complexity that hinders the deployment of them on edge devices. To achieve high throughput on hardware and preserve the model accuracy simultaneously, we propose VAQF, a framework that builds inference accelerators on FPGA platforms for quantized ViTs with binary weights and low-precision activations. Given the model structure and the desired frame rate, VAQF will automatically output the required quantization precision for activations as well as the optimized parameter settings of the accelerator that fulfill the hardware requirements. The implementations are developed with Vivado High-Level Synthesis (HLS) on the Xilinx ZCU102 FPGA board, and the evaluation results with the DeiT-base model indicate that a frame rate requirement of 24 frames per second (FPS) is satisfied with 8-bit activation quantization, and a target of 30 FPS is met with 6-bit activation quantization. To the best of our knowledge, this is the first time quantization has been incorporated into ViT acceleration on FPGAs with the help of a fully automatic framework to guide the quantization strategy on the software side and the accelerator implementations on the hardware side given the target frame rate. Very small compilation time cost is incurred compared with quantization training, and the generated accelerators show the capability of achieving real-time execution for state-of-the-art ViT models on FPGAs.

show abstract

CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

Xie

Zhang

Shen

et al. 2021

Lecture Notes in Computer Science

Self Cite

359

177

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have been the de facto standard for nowadays 3D medical image segmentation. The convolutional operations used in these networks, however, inevitably have limitations in modeling the long-range dependency due to their inductive bias of locality and weight sharing. Although Transformer was born to address this issue, it suffers from extreme computational and spatial complexities in processing high-resolution 3D feature maps. In this paper, we propose a novel framework that efficiently bridges a Convolutional neural network and a Transformer (CoTr) for accurate 3D medical image segmentation. Under this framework, the CNN is constructed to extract feature representations and an efficient deformable Transformer (DeTrans) is built to model the long-range dependency on the extracted feature maps. Different from the vanilla Transformer which treats all image positions equally, our DeTrans pays attention only to a small set of key positions by introducing the deformable self-attention mechanism. Thus, the computational and spatial complexities of DeTrans have been greatly reduced, making it possible to process the multi-scale and highresolution feature maps, which are usually of paramount importance for image segmentation. We conduct an extensive evaluation on the Multi-Atlas Labeling Beyond the Cranial Vault (BCV) dataset that covers 11 major human organs. The results indicate that our CoTr leads to a substantial performance improvement over other CNN-based, transformerbased, and hybrid methods on the 3D multi-organ segmentation task.

show abstract

End-to-End Video Instance Segmentation with Transformers

Cited by 56 publications

References 27 publications

TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers

TransVOD: End-to-end Video Object Detection with Spatial-Temporal Transformers

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

Contact Info

Product

Resources

About