Video Propagation Networks

Abstract

We propose a technique that propagates information forward through video data. The method is conceptually simple and can be applied to tasks that require the propagation of structured information, such as semantic labels, based on video content. We propose a Video Propagation Network that processes video frames in an adaptive manner. The model is applied online: it propagates information forward without the need to access future frames. In particular we combine two components, a temporal bilateral network for dense and video adaptive filtering, followed by a spatial network to refine features and increased flexibility. We present experiments on video object segmentation and semantic video segmentation and show increased performance comparing to the best previous task-specific methods, while having favorable runtime. Additionally we demonstrate our approach on an example regression task of color propagation in a grayscale video. arXiv:1612.05478v3 [cs.CV] 11 Apr 2017 processing: General applicability: VPNs can be used to propagate any type of information content i.e., both discrete (e.g., semantic labels) and continuous (e.g., color) information across video frames. Online propagation: The method needs no future frames and can be used for online video analysis. Long-range and image adaptive: VPNs can efficiently handle a large number of input frames and are adaptive to the video with long-range pixel connections. End-to-end trainable: VPNs can be trained end-to-end, so they can be used in other deep network architectures. Favorable runtime: VPNs have favorable runtime in comparison to many current best methods, what makes them amenable for learning with large datasets.Empirically we show that VPNs, despite being generic, perform better than published approaches on video object segmentation and semantic label propagation while being faster. VPNs can easily be integrated into sequential perframe approaches and require only a small fine-tuning step that can be performed separately.

See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks

“…gating Frame Pair one-shot learning framework [5,59], or used a maskpropagation network [25]. In addition, both object tracking [29,8,12,36] and person re-identification [34,66] have been fused into SVOS task to handle deformation and occlusion issues.…”

LIP: Learning Instance Propagation for Video Object Segmentation

“…It has created a large number of synthetic video training data from Pascal VOC [11,12], ECSSD [49] and MSRA10K [7] DAVIS 2017 benchmark, we exclude PReMVOS [38] and OSVOS+ [39] as they both use multiple specialized networks in multiple processes to refine their results. For DAVIS 2016, we compare with OnAVOS [52], FAVOS [5], OSVOS [3], MSK [42], PML [4], SFL [6], OSMN [57], CTN [27] and VPN [26]. We detect multiple objects and evaluate in the way for single-object.…”

“…VPN [26], MSK [42] and RGMP [54] learn to propagate mask for the VOS task. VPN utilizes learnable bilateral filters to achieve video-adaptive information propagation across frames.…”

RANet: Ranking Attention Network for Fast Video Object Segmentation

“…For single object VOS, we compare our RANet with 6 state-of-the-art OL based and 11 offline methods [1, 3, 8-10, 19, 22, 23, 35, 37, 38, 40, 45, 49-51, 59] in Table 1, including OSVOS-S [37], PReMVOS [35], RGMP [38], FEELVOS [49], etc. To evaluate our RANet trained with static images, we compare it with some methods [22,23,36,40,47] without using DAVIS training set. For multi-object VOS, we compare with some state-of-theart offline methods [3,9,19,50,59], and also list results of some OL based methods [1,3,19,37,50] for reference.…”