Interactive Video Object Segmentation Using Global and Local Transfer Modules

Abstract

An interactive video object segmentation algorithm, which takes scribble annotations on query objects as input, is proposed in this paper. We develop a deep neural network, which consists of the annotation network (A-Net) and the transfer network (T-Net). First, given user scribbles on a frame, A-Net yields a segmentation result based on the encoder-decoder architecture. Second, T-Net transfers the segmentation result bidirectionally to the other frames, by employing the global and local transfer modules. The global transfer module conveys the segmentation information in an annotated frame to a target frame, while the local transfer module propagates the segmentation information in a temporally adjacent frame to the target frame. By applying A-Net and T-Net alternately, a user can obtain desired segmentation results with minimal efforts. We train the entire network in two stages, by emulating user scribbles and employing an auxiliary loss. Experimental results demonstrate that the proposed interactive video object segmentation algorithm outperforms the state-of-the-art conventional algorithms. Codes and models are available at https://github.com/yuk6heo/IVOS-ATNet.

Fast Interactive Video Object Segmentation with Graph Neural Networks

“…Oh et al [38] measure a 44% increase in error rate when extra training data is omitted, indicating that methods with many parameters are data-hungry and underperform without the help of additional data. Method AUC J AUC J &F Extra data Heo et al [39] 0.771 0.809 Heo et al [47] 0.704 -Oh et al [38] 0.691 -Miao et al [40] 0.749 --Oh et al [48] 0.702 --Oh et al [38] 0 In order to gain some insight into the data structure, we show the error rate of prediction in case of individual videos in Fig. 4 after the second and the eighth interaction steps.…”

“…A slight deficiency of their method is the lack of weight sharing between the two networks in the convolutional layers. Heo et al achieve superior results with their feature information transfer modules [39]. A drawback of their method is the need to use multiple additional segmentation datasets for their training process.…”

“…Miao et al proposes a more efficient solution by computing all feature representations during the preprocessing stage and only utilizing shallow networks during prediction [40]. They implement global and local memory modules in a somewhat similar fashion to [39].…”