CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing

Abstract

In this work we propose a capsule-based approach for semi-supervised video object segmentation. Current video object segmentation methods are frame-based and often require optical flow to capture temporal consistency across frames which can be difficult to compute. To this end, we propose a video based capsule network, CapsuleVOS, which can segment several frames at once conditioned on a reference frame and segmentation mask. This conditioning is performed through a novel routing algorithm for attention-based efficient capsule selection. We address two challenging issues in video object segmentation: 1) segmentation of small objects and 2) occlusion of objects across time. The issue of segmenting small objects is addressed with a zooming module which allows the network to process small spatial regions of the video. Apart from this, the framework utilizes a novel memory module based on recurrent networks which helps in tracking objects when they move out of frame or are occluded. The network is trained end-to-end and we demonstrate its effectiveness on two benchmark video object segmentation datasets; it outperforms current offline approaches on the Youtube-VOS dataset while having a run-time that is almost twice as fast as competing methods. The code is publicly available at https://github.com/KevinDuarte/CapsuleVOS.

Capsule-based Object Tracking with Natural Language Specification

“…The implementation of our capsule architecture is based on [19]. Given its advances, researchers have introduced some applications of CapsNet for different vision tasks [12,24,30,33]. In this paper, we introduce the capsule network into tracking by with natural-language specification.…”

“…Then, these capsules are routed via a dynamic routing algorithm that considers the agreement between capsules, thus forming meaningful spatial relationships not found in standard CNNs. Further, CapsNet has achieved competitive results in multiple video tasks [11,12,33]. Depending on the above observations, we apply the CapsNet to visual object tracking in this paper.…”

DPR-CAE: Capsule Autoencoder with Dynamic Part Representation for Image Parsing

“…Capsule networks are designed to parse an image into a hierarchy of objects, parts and relations. [Hinton et al, 2011] first introduced the idea of capsule networks. [Sabour et al, 2017] revisited the capsule concept and introduced capsule framework for image classification .…”

“…[Sabour et al, 2017] revisited the capsule concept and introduced capsule framework for image classification . Subsequent works are produced to improve the routing algorithms [Hinton et al, 2018;Taeyoung et al, 2019] and scale to other tasks [Duarte et al, 2019;Xiang et al, 2020;.…”

“…Capsule networks [Hinton et al, 2011;Sabour et al, 2017] are designed to understand images, which parse an image into a hierarchy of objects, parts and relations. Studies show that infants form early object perception by dividing visual inputs into simple part units in an unsupervised way.…”