Deformable Sprites for Unsupervised Video Decomposition

Ye, Vickie; Li, Zhengqi; Tucker, Richard; Kanazawa, Angjoo; Snavely, Noah

doi:10.1109/cvpr52688.2022.00268

Cited by 40 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our encoder network is a ResNet-32-CIFAR10 [20], that is truncated after layer 3 with a Gaussian feature pooling described in supplementary material. For our unsupervised experiments, we use as generator g θ the U-Net architecture of Deformable Sprites [49] which converged quickly, and for our supervised experiments a 2-layer MLP similar to MarioNette [41] which produces sprites of higher quality. The networks π θ and p θ are a single linear layers followed by layer-normalization.…”

Section: Losses and Training Detailsmentioning

confidence: 99%

The Learnable Typewriter: A Generative Approach to Text Line Analysis

Siglidis¹,

Gonthier²,

Gaubil³

et al. 2023

Preprint

View full text Add to dashboard Cite

We present a generative document-specific approach to character analysis and recognition in text lines. Our main idea is to build on unsupervised multi-object segmentation methods and in particular those that reconstruct images based on a limited amount of visual elements, called sprites. Our approach can learn a large number of different characters and leverage line-level annotations when available. Our contribution is twofold. First, we provide the first adaptation and evaluation of a deep unsupervised multi-object segmentation approach for text line analysis. Since these methods have mainly been evaluated on synthetic data in a completely unsupervised setting, demonstrating that they can be adapted and quantitatively evaluated on real text images and that they can be trained using weak supervision are significant progresses. Second, we demonstrate the potential of our method for new applications, more specifically in the field of paleography, which studies the history and variations of handwriting, and for cipher analysis. We evaluate our approach on three very different datasets: a printed volume of the Google1000 dataset [19,46], the Copiale cipher [2, 27] and historical handwritten charters from the 12th and early 13th century [6,44].

show abstract

Section: Losses and Training Detailsmentioning

confidence: 99%

The Learnable Typewriter: A Generative Approach to Text Line Analysis

Siglidis¹,

Gonthier²,

Gaubil³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The predominant way to identify the objects present in a scene is to segment twodimensional images using extensive manual annotation (Kirillov et al, 2023;Wang et al, 2023a), but relying on human supervision introduces challenges and scales poorly to 3D data. As an alternative, an extensive line of work on unsupervised object discovery (Russell et al, 2006;Rubinstein et al, 2013;Oktay et al, 2018;Hénaff et al, 2022;Smith et al, 2022;Ye et al, 2022;Monnier et al, 2023) proposes different inductive biases (Locatello et al, 2019) that encourage awareness of objects in a scene. However, these approaches are largely restricted to either 2D images or constrained 3D data (Yu et al, 2021;Sajjadi et al, 2022), limiting their applicability to complex 3D scenes.…”

Section: Related Workmentioning

confidence: 99%

BlobGAN: Spatially Disentangled Scene Representations

Epstein

Park

Zhang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We introduce a method to generate 3D scenes that are disentangled into their component objects. This disentanglement is unsupervised, relying only on the knowledge of a large pretrained textto-image model. Our key insight is that objects can be discovered by finding parts of a 3D scene that, when rearranged spatially, still produce valid configurations of the same scene. Concretely, our method jointly optimizes multiple NeRFs from scratch-each representing its own object-along with a set of layouts that composite these objects into scenes. We then encourage these composited scenes to be in-distribution according to the image generator. We show that despite its simplicity, our approach successfully generates 3D scenes decomposed into individual objects, enabling new capabilities in text-to-3D content creation. See our project page for results and an interactive demo: https://dave.ml/layoutlearning/

show abstract

“…In general, many popular algorithms [4], [6]- [17] require supervised training on large-scale datasets to obtain the segmentation masks. Alternatively, a number of works [25]- [27] based on the offline setting employ a deep neural network to discover the objects of interest from the perspective of completely unsupervised concepts in the traditional methods. Lu et al [67] proposed a unified framework for unsupervised learning, aimed at object segmentation through the exploitation of the inherent consistency across adjacent frames in unlabeled videos.…”

Section: Related Workmentioning

confidence: 99%

“…DyStaB employs static and dynamic models to learn object saliency from motion in a video, which can then be applied at inference time to segment objects, even in static images [25]. Deformable Sprites (DeSprites) [27] are a type of video autoencoder model that is optimized on each individual video. Our work also optimizes an auto-encoder on a specific sequence in an unsupervised manner.…”

Section: Related Workmentioning

confidence: 99%

“…However, traditional physics-optimizationbased approaches incur significant computational costs due to the optimization process over the entire video. Unsupervised methods [25]- [27] based on deep neural networks have gained significant advantage by learning a deep representation on a video dataset. Although those methods achieve good performance, they cannot handle streaming video because they work offline and require the entire video to be processed before making predictions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Implicit Motion-Compensated Network for Unsupervised Video Object Segmentation

Lin

Chen

et al. 2022

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Online unsupervised video object segmentation (UVOS) uses the previous frames as its input to automatically separate the primary object(s) from a streaming video without using any further manual annotation. A major challenge is that the model has no access to the future and must rely solely on the history, i.e., the segmentation mask is predicted from the current frame as soon as it is captured. In this work, a novel contrastive motion clustering algorithm with an optical flow as its input is proposed for the online UVOS by exploiting the common fate principle that visual elements tend to be perceived as a group if they possess the same motion pattern. We build a simple and effective auto-encoder to iteratively summarize non-learnable prototypical bases for the motion pattern, while the bases in turn help learn the representation of the embedding network. Further, a contrastive learning strategy based on a boundary prior is developed to improve foreground and background feature discrimination in the representation learning stage. The proposed algorithm can be optimized on arbitrarily-scale data (i.e., frame, clip, dataset) and performed in an online fashion. Experiments on DAVIS16, FBMS, and SegTrackV2 datasets show that the accuracy of our method surpasses the previous state-of-the-art (SoTA) online UVOS method by a margin of 0.8%, 2.9%, and 1.1%, respectively. Furthermore, by using an online deep subspace clustering to tackle the motion grouping, our method is able to achieve higher accuracy at 3× faster inference time compared to SoTA online UVOS method, and making a good trade-off between effectiveness and efficiency.

show abstract

Deformable Sprites for Unsupervised Video Decomposition

Cited by 40 publications

References 26 publications

The Learnable Typewriter: A Generative Approach to Text Line Analysis

The Learnable Typewriter: A Generative Approach to Text Line Analysis

BlobGAN: Spatially Disentangled Scene Representations

Implicit Motion-Compensated Network for Unsupervised Video Object Segmentation

Contact Info

Product

Resources

About