Learning What to Learn for Video Object Segmentation

Abstract

Video object segmentation (VOS) is a highly challenging problem, since the target object is only defined during inference with a given first-frame reference mask. The problem of how to capture and utilize this limited target information remains a fundamental research question. We address this by introducing an end-to-end trainable VOS architecture that integrates a differentiable few-shot learning module. This internal learner is designed to predict a powerful parametric model of the target by minimizing a segmentation error in the first frame. We further go beyond standard few-shot learning techniques by learning what the few-shot learner should learn. This allows us to achieve a rich internal representation of the target in the current frame, significantly increasing the segmentation accuracy of our approach. We perform extensive experiments on multiple benchmarks. Our approach sets a new state-ofthe-art on the large-scale YouTube-VOS 2018 dataset by achieving an overall score of 81.5, corresponding to a 2.6% relative improvement over the previous best result.

Discriminative correlation filters in robot vision

“…This extends paper D (and paper [2]) to improve the robustness of the correlationfilter model, by additionally modeling distracting objects. The author initiated the project, contributed to the idea development, implemen tation, experiment design and execution and writing.…”

“…At 76 percent, our method is approximately 7 percentage points below STM, but is significantly faster at 22 frames per second. Paper D was subsequently extended into the learningwhattolearn (LWL) method [2] for video object segmentation. The main feature of LWL is that, unlike the method in paper D, it can backpropagate through the target model trainer.…”

“…Finally, we can place papers D and E in context to the stateoftheart, figure 6.1 shows the improvement over time, of mean J and F score improvement. The methods were evaluated on the DAVIS 2017 validation split, and includes both the methods mentioned before, i.e OSVOS [6], RGMP [26], STM [27] and LWL [2], as well as the more recent method CFBI [41]. Both STM and CFBI were considered stateoftheart at the time of writing.…”

“…This separation allows imposing spe cial attention to challenging regions that are most likely to degrade the per formance. We demonstrate the prominence of this formulation by modifying the learningwhattolearn [2] method to be distractoraware. Our proposed approach sets a new stateoftheart on the DAVIS val dataset, and improves over the baseline on the DAVIS testdev benchmark by 4.8 percentage points.…”

Learning Representations for Segmentation and Registration

“…Typically, these layers comprise simple operators like convolutional filters, standard linear operators, point-wise non-linear activation functions, or self-attention modules [110], but they compose a highly descriptive and powerful function. We further note that the layers can also include more complex processes and even optimization algorithms [41,5,6].…”