GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement

Engelcke, Martin; Jones, Ōiwi Parker; Posner, Ingmar

doi:10.48550/arxiv.2104.09958

Cited by 10 publications

(19 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparison to prior work. In the appendix, we further compare our approach with two recent end-to-end object-centric models (Lin et al, 2020b;Engelcke et al, 2021). Both models struggle to scale to the FISHBOWL dataset, highlighting the advantage of our multi-stage approach compared to end-to-end scene models.…”

Section: Methodsmentioning

confidence: 99%

“…Those approaches typically use additional supervision such as ground-truth segmentation or additional views, with Ehrhardt et al (2020); Niemeyer & Geiger (2021) being notable exceptions. While most methods can decompose a given scene into its constituent objects, only few are fully-generative in the sense that they can generate novel scenes (Lin et al, 2020a;Ehrhardt et al, 2020;von Kügelgen et al, 2020;Engelcke et al, 2021;Niemeyer & Geiger, 2021;Dittadi & Winther, 2019).…”

Section: Related Workmentioning

confidence: 99%

“…Object-centric approaches aim to capture this compositionality and have been considerably improved over the past few years (e.g., Locatello et al 2020;Engelcke et al 2021). However, these models tend to be difficult to train and do not yet scale well to visually more complex scenes.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Object Learning via Common Fate

Tangemann,

Schneider,

von Kügelgen

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning generative object models from unlabelled videos is a long standing problem and required for causal scene modeling. We decompose this problem into three easier subtasks, and provide candidate solutions for each of them. Inspired by the Common Fate Principle of Gestalt Psychology, we first extract (noisy) masks of moving objects via unsupervised motion segmentation. Second, generative models are trained on the masks of the background and the moving objects, respectively. Third, background and foreground models are combined in a conditional "dead leaves" scene model to sample novel scene configurations where occlusions and depth layering arise naturally. To evaluate the individual stages, we introduce the FISHBOWL dataset positioned between complex real-world scenes and common object-centric benchmarks of simplistic objects. We show that our approach allows learning generative models that generalize beyond the occlusions present in the input videos, and represent scenes in a modular fashion that allows sampling plausible scenes outside the training distribution by permitting, for instance, object numbers or densities not observed in the training set.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Object Learning via Common Fate

Tangemann,

Schneider,

von Kügelgen

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…There are many parametric forms M could employ. For instance, one could consider generating multiple masks sequentially in an auto-regressive manner as seen in Engelcke et al (2021). However, we find that the simplest setup suffices, where M : R c×w×h → R N ×w×h consists of a learnable neural network and a pixelwise softmax layer σ applied across N masks to ensure that the sum of a given pixel across all the masks equals 1.…”

Section: Occlusion Model Mmentioning

confidence: 99%

Adversarial Masking for Self-Supervised Learning

Shi¹,

Siddharth²,

Torr³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose ADIOS, a masked image modeling (MIM) framework for self-supervised learning, which simultaneously learns a masking function and an image encoder using an adversarial objective. The image encoder is trained to minimise the distance between representations of the original and that of a masked image. The masking function, conversely, aims at maximising this distance. ADIOS consistently improves on state-ofthe-art self-supervised learning (SSL) methods on a variety of tasks and datasets-including classification on ImageNet100 and STL10, transfer learning on CIFAR10/100, Flowers102 and iNaturalist, as well as robustness evaluated on the backgrounds challenge (Xiao et al., 2021)-while generating semantically meaningful masks. Unlike modern MIM models such as MAE, BEiT and iBOT, ADIOS does not rely on the image-patch tokenisation construction of Vision Transformers, and can be implemented with convolutional backbones. We further demonstrate that the masks learned by ADIOS are more effective in improving representation learning of SSL methods than masking schemes used in popular MIM models.

show abstract

“…Datasets We perform a qualitative evaluation using the realworld Sketchy dataset [23], which contains demonstration trajectories of a robot arm performing different tasks involving a set of objects. The images are pre-processed as in [37], using a 128 × 128 resolution and sequences of length 10. Sketchy, however, does not contain object annotations, which prohibits the quantitative evaluation of object segmentation and tracking methods.…”

Section: Learningmentioning

confidence: 99%

APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation

Jones

Engelcke

et al. 2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Self Cite

View full text Add to dashboard Cite

Recent advances in unsupervised learning for object detection, segmentation, and tracking hold significant promise for applications in robotics. A common approach is to frame these tasks as inference in probabilistic latent-variable models. In this paper, however, we show that the current state-of-the-art struggles with visually complex scenes such as typically encountered in robot manipulation tasks. We propose APEX, a new latent-variable model which is able to segment and track objects in more realistic scenes featuring objects that vary widely in size and texture, including the robot arm itself. This is achieved by a principled mask normalisation algorithm and a high-resolution scene encoder. To evaluate our approach, we present results on the real-world Sketchy dataset. This dataset, however, does not contain ground truth masks and object IDs for a quantitative evaluation. We thus introduce the Panda Pushing Dataset (P2D) which shows a Panda arm interacting with objects on a table in simulation and which includes groundtruth segmentation masks and object IDs for tracking. In both cases, APEX comprehensively outperforms the current state-ofthe-art in unsupervised object segmentation and tracking. We demonstrate the efficacy of our segmentations for robot skill execution on an object arrangement task, where we also achieve the best or comparable performance among all the baselines.

show abstract

GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement

Cited by 10 publications

References 32 publications

Unsupervised Object Learning via Common Fate

Unsupervised Object Learning via Common Fate

Adversarial Masking for Self-Supervised Learning

APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation

Contact Info

Product

Resources

About