STEm-Seg: Spatio-Temporal Embeddings for Instance Segmentation in Videos

Athar, Ali; Mahadevan, Sabarinath; Ošep, Aljoša; Leal-Taixé, Laura; Leibe, Bastian

doi:10.1007/978-3-030-58621-8_10

Cited by 142 publications

(177 citation statements)

References 76 publications

Supporting

Mentioning

177

Contrasting

Order By: Relevance

“…Such scenarios usually require the algorithm to be more robust. As the test set evaluation server is closed, we followed most previous works [ 3 , 10 , 11 , 13 , 14 ] and evaluated our method on the validation set. The evaluation metrics are Average Precision (

) calculated based on multiple intersection-over-union (IoU) thresholds and Average Recall (

) defined as the maximum recall given some fixed number of segmented instances per video.…”

Section: Resultsmentioning

confidence: 99%

“…The gap of accuracy between our method and MaskProp [ 11 ] is mainly due to the fact that Maskprop combines multiple networks and post-processing strategies which are actually time-consuming. For STEm-Seg [ 13 ] and VisTR [ 14 ], both methods process the entire video sequence at the same time. Such methods usually impose limits on video resolution and length, which are difficult to extend to online applications.…”

Section: Resultsmentioning

confidence: 99%

“…IPDT [ 12 ] proposes a framework which uses instance-level and pixel-level embeddings in both tracking directions to calibrate segmentation results. STEm-Seg [ 13 ] models a video sequence as a 3D spatial-temporal volume. The network consists of a 3D CNN based encoder-decoder to process the backbone feature maps.…”

Section: Related Workmentioning

confidence: 99%

“…Previous works [ 3 , 10 , 11 , 12 , 13 , 14 ] mostly rely on two strategies. On one hand, some methods use complex post-processing to associate frame level instances.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Instance Sequence Queries for Video Instance Segmentation with Transformers

Xu¹,

Vivet²

2021

Sensors

View full text Add to dashboard Cite

Existing methods for video instance segmentation (VIS) mostly rely on two strategies: (1) building a sophisticated post-processing to associate frame level segmentation results and (2) modeling a video clip as a 3D spatial-temporal volume with a limit of resolution and length due to memory constraints. In this work, we propose a frame-to-frame method built upon transformers. We use a set of queries, called instance sequence queries (ISQs), to drive the transformer decoder and produce results at each frame. Each query represents one instance in a video clip. By extending the bipartite matching loss to two frames, our training procedure enables the decoder to adjust the ISQs during inference. The consistency of instances is preserved by the corresponding order between query slots and network outputs. As a result, there is no need for complex data association. On TITAN Xp GPU, our method achieves a competitive 34.4% mAP at 33.5 FPS with ResNet-50 and 35.5% mAP at 26.6 FPS with ResNet-101 on the Youtube-VIS dataset.

show abstract

) calculated based on multiple intersection-over-union (IoU) thresholds and Average Recall (

) defined as the maximum recall given some fixed number of segmented instances per video.…”

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…Previous works [ 3 , 10 , 11 , 12 , 13 , 14 ] mostly rely on two strategies. On one hand, some methods use complex post-processing to associate frame level instances.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Instance Sequence Queries for Video Instance Segmentation with Transformers

Xu¹,

Vivet²

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Our method also has other advantages compared to the other methods. Unlike object detection and tracking methods [12], [21], [31], which require prior knowledge of known objects (object proposals from pre-trained Imagenet models) to solve UVOS accurately, our method does not depend on any prior knowledge of the segmented objects (as we only use affinities to perform UVOS). Hence our model is capable of generalizing well to unseen object classes.…”

Section: ) Davis16mentioning

confidence: 99%

Unsupervised video object segmentation: An affinity and edge learning approach.

Maruthamuthu¹,

Tennakoon²,

Hoseinnezhad³

et al. 2021

Preprint

View full text Add to dashboard Cite

<div>This paper presents a new approach to solve unsupervised video object segmentation~(UVOS) problem (called TMNet). The UVOS is still a challenging problem as prior methods suffer from issues like generalization errors to segment multiple objects in unseen test videos (category agnostic), over reliance on inaccurate optic flow, and problem towards capturing fine details at object boundaries. These issues make the UVOS, particularly in presence of multiple objects, an ill-defined problem. Our focus is to constrain the problem and improve the segmentation results by inclusion of multiple available cues such as appearance, motion, image edge, flow edge and tracking information through neural attention. To solve the challenging category agnostic multiple object UVOS, our model is designed to predict neighbourhood affinities for being part of the same object and cluster those to obtain accurate segmentation. To achieve multi cue based neural attention, we designed a Temporal Motion Attention module, as part of our segmentation framework, to learn the spatio-temporal features. To refine and improve the accuracy of object segmentation boundaries, an edge refinement module (using image and optic flow edges) and a geometry based loss function are incorporated. The overall framework is capable of segmenting and finding accurate objects' boundaries without any heuristic post processing. This enables the method to be used for unseen videos. Experimental results on challenging DAVIS16 and multi object DAVIS17 datasets shows that our proposed TMNet performs favourably compared to the state-of-the-art methods without post processing.</div>

show abstract

Seabird surveillance: combining CCTV and artificial intelligence for monitoring and research

Hentati‐Sundberg

Olin

Reddy

et al. 2023

Remote Sens Ecol Conserv

View full text Add to dashboard Cite

Ecological research and monitoring need to be able to rapidly convey information that can form the basis of scientifically sound management. Automated sensor systems, especially if combined with artificial intelligence, can contribute to such rapid high‐resolution data retrieval. Here, we explore the prospects of automated methods to generate insights for seabirds, which are often monitored for their high conservation value and for being sentinels for marine ecosystem changes. We have developed a system of video surveillance combined with automated image processing, which we apply to common murres Uria aalge. The system uses a deep learning algorithm for object detection (YOLOv5) that has been trained on annotated images of adult birds, chicks and eggs, and outputs time, location, size and confidence level of all detections, frame‐by‐frame, in the supplied video material. A total of 144 million bird detections were generated from a breeding cliff over three complete breeding seasons (2019–2021). We demonstrate how object detection can be used to accurately monitor breeding phenology and chick growth. Our automated monitoring approach can also identify and quantify rare events that are easily missed in traditional monitoring, such as disturbances from predators. Further, combining automated video analysis with continuous measurements from a temperature logger allows us to study impacts of heat waves on nest attendance in high detail. Our automated system thus produces comparable, and in several cases significantly more detailed, data than those generated from observational field studies. By running in real time on the camera streams, it has the potential to supply researchers and managers with high‐resolution up‐to‐date information on seabird population status. We describe how the system can be modified to fit various types of ecological research and monitoring goals and thereby provide up‐to‐date support for conservation and ecosystem management.

show abstract

STEm-Seg: Spatio-Temporal Embeddings for Instance Segmentation in Videos

Cited by 142 publications

References 76 publications

Instance Sequence Queries for Video Instance Segmentation with Transformers

Instance Sequence Queries for Video Instance Segmentation with Transformers

Unsupervised video object segmentation: An affinity and edge learning approach.

Seabird surveillance: combining CCTV and artificial intelligence for monitoring and research

Contact Info

Product

Resources

About