Enhanced Memory Network for Video Segmentation

Zhou, Zefeng; Ren, Lejian; Xiong, Pengfei; Ji, Yifei; Wang, Peisen; Fan, Haoqiang; Liu, Si

doi:10.1109/iccvw.2019.00083

Cited by 24 publications

(13 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 5 tabulates the results. Our base model beats the previous winner [28], and our ensemble model achieves top-1 on the still active validation leaderboard at the time of writing. The 2021 challenge has not released the official results/technical reports by NeurIPS 2021 deadline, but our method outperforms all of them on the validation set.…”

Section: Multi-scale Testing and Ensemblementioning

confidence: 74%

“…They usually represent the highest achievable performance at the time. On the latest YouTubeVOS 2019 validation split [71], our base model (84.2 G) outperforms the previous challenge winner [28] (based on STM [17], 82.0 G) by a large margin. With ensemble and multi-scale testing (details in the supplementary material), our method is ranked first place (86.7 G) at the time of submission on the still active leaderboard.…”

Section: Leaderboard Resultsmentioning

confidence: 91%

“…Most current methods either fit a model using the initial segmentation [4,5,6,7,8] or leverage temporal propagation [9,10,11,12,13,14,15], particularly with spatio-temporal matching [16,17,18,19,20,21,22,23]. Space-Time Memory networks [17] are especially popular recently due to its high performance and simplicity -many variants [21,15,22,20,23,24,25,26], including competitions' winners [27,28], have been developed to improve the speed, reduce memory usage, or to regularize the memory readout process of STM.…”

Section: Introductionmentioning

confidence: 99%

“…MS denotes multi-scale testing. EMN[28] is the previous challenge winner. Methods are ranked by performance, with variants of the same method grouped together.The Adam[64] optimizer is used with default momentum β 1 = 0.9, β 2 = 0.999, a base learning rate of 10 −5 , and a L2 weight decay of 10 −7 .…”

mentioning

confidence: 99%

See 3 more Smart Citations

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Cheng¹,

Tai²,

Tang³

2021

Preprint

View full text Add to dashboard Cite

This paper presents a simple yet effective approach to modeling space-time correspondences in the context of video object segmentation. Unlike most existing approaches, we establish correspondences directly between frames without reencoding the mask features for every object, leading to a highly efficient and robust framework. With the correspondences, every node in the current query frame is inferred by aggregating features from the past in an associative fashion. We cast the aggregation process as a voting problem and find that the existing inner-product affinity leads to poor use of memory with a small (fixed) subset of memory nodes dominating the votes, regardless of the query. In light of this phenomenon, we propose using the negative squared Euclidean distance instead to compute the affinities. We validated that every memory node now has a chance to contribute, and experimentally showed that such diversified voting is beneficial to both memory efficiency and inference accuracy. The synergy of correspondence networks and diversified voting works exceedingly well, achieves new state-of-the-art results on both DAVIS and YouTubeVOS datasets while running significantly faster at 20+ FPS for multiple objects without bells and whistles.

show abstract

Section: Multi-scale Testing and Ensemblementioning

confidence: 74%

Section: Leaderboard Resultsmentioning

confidence: 91%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Cheng¹,

Tai²,

Tang³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Memory networks have been introduced to enhance the reasoning ability of the model in VideoQA [8,9] and video object segmentation [10,11,12], but have never been introduced in video semantic segmentation as we know. [8] uses episodic memory to conduct multiple cycles of inference by interacting the question with video features conditioned on current memory.…”

Section: Introductionmentioning

confidence: 99%

Temporal Memory Attention for Video Semantic Segmentation

Wang

Liu

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Video semantic segmentation requires to utilize the complex temporal relations between frames of the video sequence. Previous works usually exploit accurate optical flow to leverage the temporal relations, which suffer much from heavy computational cost. In this paper, we propose a Temporal Memory Attention Network (TMANet) to adaptively integrate the long-range temporal relations over the video sequence based on the self-attention mechanism without exhaustive optical flow prediction. Specially, we construct a memory using several past frames to store the temporal information of the current frame. We then propose a temporal memory attention module to capture the relation between the current frame and the memory to enhance the representation of the current frame. Our method achieves new state-of-theart performances on two challenging video semantic segmentation datasets, particularly 80.3% mIoU on Cityscapes and 76.5% mIoU on CamVid with ResNet-50.

show abstract