Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation

Bertasius, Gedas; Torresani, Lorenzo

doi:10.1109/cvpr42600.2020.00976

Cited by 170 publications

(184 citation statements)

References 32 publications

Supporting

Mentioning

184

Contrasting

Order By: Relevance

“…Such scenarios usually require the algorithm to be more robust. As the test set evaluation server is closed, we followed most previous works [ 3 , 10 , 11 , 13 , 14 ] and evaluated our method on the validation set. The evaluation metrics are Average Precision (

) calculated based on multiple intersection-over-union (IoU) thresholds and Average Recall (

) defined as the maximum recall given some fixed number of segmented instances per video.…”

Section: Resultsmentioning

confidence: 99%

“…We also slightly outperform SipMask [ 3 ] in both

and runtime. The gap of accuracy between our method and MaskProp [ 11 ] is mainly due to the fact that Maskprop combines multiple networks and post-processing strategies which are actually time-consuming. For STEm-Seg [ 13 ] and VisTR [ 14 ], both methods process the entire video sequence at the same time.…”

Section: Resultsmentioning

confidence: 99%

“…Similar to the Mask-Track RCNN [ 10 ], the model is extended with a tracking feature branch for the VIS task. Maskprop [ 11 ] uses predicted masks to crop image features, and to propagate the instance-specific features to improve the final results. IPDT [ 12 ] proposes a framework which uses instance-level and pixel-level embeddings in both tracking directions to calibrate segmentation results.…”

Section: Related Workmentioning

confidence: 99%

“…Previous works [ 3 , 10 , 11 , 12 , 13 , 14 ] mostly rely on two strategies. On one hand, some methods use complex post-processing to associate frame level instances.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Instance Sequence Queries for Video Instance Segmentation with Transformers

Xu¹,

Vivet²

2021

Sensors

View full text Add to dashboard Cite

Existing methods for video instance segmentation (VIS) mostly rely on two strategies: (1) building a sophisticated post-processing to associate frame level segmentation results and (2) modeling a video clip as a 3D spatial-temporal volume with a limit of resolution and length due to memory constraints. In this work, we propose a frame-to-frame method built upon transformers. We use a set of queries, called instance sequence queries (ISQs), to drive the transformer decoder and produce results at each frame. Each query represents one instance in a video clip. By extending the bipartite matching loss to two frames, our training procedure enables the decoder to adjust the ISQs during inference. The consistency of instances is preserved by the corresponding order between query slots and network outputs. As a result, there is no need for complex data association. On TITAN Xp GPU, our method achieves a competitive 34.4% mAP at 33.5 FPS with ResNet-50 and 35.5% mAP at 26.6 FPS with ResNet-101 on the Youtube-VIS dataset.

show abstract

) calculated based on multiple intersection-over-union (IoU) thresholds and Average Recall (

) defined as the maximum recall given some fixed number of segmented instances per video.…”

Section: Resultsmentioning

confidence: 99%

“…We also slightly outperform SipMask [ 3 ] in both

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…Previous works [ 3 , 10 , 11 , 12 , 13 , 14 ] mostly rely on two strategies. On one hand, some methods use complex post-processing to associate frame level instances.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Instance Sequence Queries for Video Instance Segmentation with Transformers

Xu¹,

Vivet²

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Similarly, Jain et al [9] propagated features instead of segmentation with flow from key frames. A similar idea was applied to instance segmentation by Bertasius and Torresani [2]. Although these methods seem close, the task is different: (1) they heavily rely on dense annotation in time to learn interpolations explicitly while we cannot afford it due to high cost of expert annotation and difficulty in fine-grained annotation of imperfect frames and (2) they benchmark only on visible objects, while we are solely interested in objects with limited visibility.…”

Section: Related Workmentioning

confidence: 99%

Recovering the Imperfect: Cell Segmentation in the Presence of Dynamically Localized Proteins

Çiçek

Marrakchi

Antwi

et al. 2020

Interpretable and Annotation-Efficient Learning for Medical Image Computing

View full text Add to dashboard Cite

Deploying off-the-shelf segmentation networks on biomedical data has become common practice, yet if structures of interest in an image sequence are visible only temporarily, existing frame-by-frame methods fail. In this paper, we provide a solution to segmentation of imperfect data through time based on temporal propagation and uncertainty estimation. We integrate uncertainty estimation into Mask R-CNN network and propagate motion-corrected segmentation masks from frames with low uncertainty to those frames with high uncertainty to handle temporary loss of signal for segmentation. We demonstrate the value of this approach over frame-by-frame segmentation and regular temporal propagation on data from human embryonic kidney (HEK293T) cells transiently transfected with a fluorescent protein that moves in and out of the nucleus over time. The method presented here will empower microscopic experiments aimed at understanding molecular and cellular function.

show abstract