Instances as Queries

Fang, Yuxin; Yang, Shusheng; Wang, Xinggang; Yu, Li; Chen, Fang; Shan, Ying; Feng, Bin; Liu, Wenyu

doi:10.1109/iccv48922.2021.00683

Cited by 231 publications

(117 citation statements)

References 29 publications

Supporting

Mentioning

117

Contrasting

Order By: Relevance

“…It started with DETR [18], using a transformer encoder-decoder and queries to detect a fixed number of objects. QueryInst [19] further adapted DETR to instance segmentation, by adding dynamic mask heads. Also based on DETR, MaskFormer [20] first segments the input image into a fixed number of masks and then classifies them.…”

Section: A Instance Segmentation Architecturesmentioning

confidence: 99%

Instance Segmentation for Autonomous Log Grasping in Forestry Operations

Fortin¹,

Gamache²,

Grondin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Wood logs picking is a challenging task to automate. Indeed, logs usually come in cluttered configurations, randomly orientated and overlapping. Recent work on log picking automation usually assume that the logs' pose is known, with little consideration given to the actual perception problem. In this paper, we squarely address the latter, using a datadriven approach. First, we introduce a novel dataset, named TimberSeg 1.0, that is densely annotated, i.e., that includes both bounding boxes and pixel-level mask annotations for logs. This dataset comprises 220 images with 2500 individually segmented logs. Using our dataset, we then compare three neural network architectures on the task of individual logs detection and segmentation; two region-based methods and one attention-based method. Unsurprisingly, our results show that axis-aligned proposals, failing to take into account the directional nature of logs, underperform with 19.03 mAP. A rotation-aware proposal method significantly improve results to 31.83 mAP. More interestingly, a Transformer-based approach, without any inductive bias on rotations, outperformed the two others, achieving a mAP of 57.53 on our dataset. Our use case demonstrates the limitations of region-based approaches for cluttered, elongated objects. It also highlights the potential of attention-based methods on this specific task, as they work directly at the pixel-level. These encouraging results indicate that such a perception system could be used to assist the operators on the short-term, or to fully automate log picking operations in the future.

show abstract

Section: A Instance Segmentation Architecturesmentioning

confidence: 99%

Instance Segmentation for Autonomous Log Grasping in Forestry Operations

Fortin¹,

Gamache²,

Grondin³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Frame-level VIS: Most video instance segmentation methods work at the frame-level fashion, a.k.a. trackingby-segmentation [6,10,11,14,16,19,24,26,29]. This paradigm produces instance segmentation frame-by-frame and achieves tracking by linking the current instance mask to the history tracklet.…”

Section: Related Workmentioning

confidence: 99%

“…Existing methods typically solve the VIS problem at either frame-level or clip-level. The frame-level methods [10,14,16,24,26,29] follow a tracking-by-segmentation paradigm, which first performs image instance segmentation and then links the current masks with history tracklets via data association as shown in Fig. 1 (a).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Video Instance Segmentation via Tracklet Query and Proposal

Wu¹,

Yarram²,

Liang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video Instance Segmentation (VIS) aims to simultaneously classify, segment, and track multiple object instances in videos. Recent clip-level VIS takes a short video clip as input each time showing stronger performance than frame-level VIS (tracking-by-segmentation), as more temporal context from multiple frames is utilized. Yet, most clip-level methods are neither end-to-end learnable nor real-time. These limitations are addressed by the recent VIS transformer (VisTR) [25] which performs VIS end-to-end within a clip. However, VisTR suffers from long training time due to its frame-wise dense attention. In addition, VisTR is not fully end-to-end learnable in multiple video clips as it requires a hand-crafted data association to link instance tracklets between successive clips. This paper proposes EfficientVIS, a fully end-to-end framework with efficient training and inference. At the core are tracklet query and tracklet proposal that associate and segment regions-of-interest (RoIs) across space and time by an iterative query-video interaction. We further propose a correspondence learning that makes tracklets linking between clips end-to-end learnable. Compared to VisTR, EfficientVIS requires 15× fewer training epochs while achieving state-of-the-art accuracy on the YouTube-VIS benchmark. Meanwhile, our method enables whole video instance segmentation in a single end-to-end pass without data association at all.

show abstract

“…The advent of automatic feature engineering fuels deep convolution neural networks (CNNs) to reach the remarkable success in a plethora of computer vision tasks, such as image classification [9,10,30,35], object detection [6,17,20], and semantic segmentation [8,34]. In the path of pursuing better performance than that of early prototypes such as VGG [21] and ResNet [9], current deep learning models [11,16,30] generally are embodied with billions of * Correspondence to: Shan You <youshan@sensetime.com>.…”

Section: Introductionmentioning

confidence: 99%

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Huang¹,

You²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Structural re-parameterization (Rep) methods achieve noticeable improvements on simple VGG-style networks. Despite the prevalence, current Rep methods simply reparameterize all operations into an augmented network, including those that rarely contribute to the model's performance. As such, the price to pay is an expensive computational overhead to manipulate these unnecessary behaviors. To eliminate the above caveats, we aim to bootstrap the training with minimal cost by devising a dynamic re-parameterization (DyRep) method, which encodes Rep technique into the training process that dynamically evolves the network structures. Concretely, our proposal adaptively finds the operations which contribute most to the loss in the network, and applies Rep to enhance their representational capacity. Besides, to suppress the noisy and redundant operations introduced by Rep, we devise a de-parameterization technique for a more compact reparameterization. With this regard, DyRep is more efficient than Rep since it smoothly evolves the given network instead of constructing an over-parameterized network. Experimental results demonstrate our effectiveness, e.g., DyRep improves the accuracy of ResNet-18 by 2.04% on ImageNet and reduces 22% runtime over the baseline. Code is available at: https://github.com/hunto/DyRep.

show abstract

Instances as Queries

Cited by 231 publications

References 29 publications

Instance Segmentation for Autonomous Log Grasping in Forestry Operations

Instance Segmentation for Autonomous Log Grasping in Forestry Operations

Efficient Video Instance Segmentation via Tracklet Query and Proposal

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Contact Info

Product

Resources

About