Comparing Reconstruction- and Contrastive-based Models for Visual Task Planning

Abstract

Learning state representations enables robotic planning directly from raw observations such as images. Most methods learn state representations by utilizing losses based on the reconstruction of the raw observations from a lowerdimensional latent space. The similarity between observations in the space of images is often assumed and used as a proxy for estimating similarity between the underlying states of the system. However, observations commonly contain task-irrelevant factors of variation which are nonetheless important for reconstruction, such as varying lighting and different camera viewpoints. In this work, we define relevant evaluation metrics and perform a thorough study of different loss functions for state representation learning. We show that models exploiting task priors, such as Siamese networks with a simple contrastive loss, outperform reconstruction-based representations in visual task planning.

Delaunay Component Analysis for Evaluation of Data Representations

“…For Q 2 , the conservative approach resulted in B = 85% due to 77 points being assigned to a component despite their wrong label c i , i ≥ 7 (thus leading to A = 0%), suggesting a moderate separation of classes. For Q 3 , we see that A = 8% and B = 28% in the conservative approach which indicates that the model was not able to recognize similar box configurations recorded from different camera view, which is aligned with the results obtained by Chamzas et al (2021). Lastly, we observe that flexible approach yielded only minor decrease in B for Q 2 and Q 3 .…”

“…Typically, a classification problem is used to evaluate either the ability of a model to recover labels of raw inputs, or the transferability of their representations to other domains, as done in state-of-the-art unsupervised representation learning methods (Chen et al, 2020b;Ermolov et al, 2021). However, in many scenarios such straightforward downstream classification task cannot be defined, for instance, because it does not represent the nature of the application or due to the scarcity of labeled data as often occurs in robotics (Chamzas et al, 2021;Lippi et al, 2020). In these scenarios, representations are commonly evaluated on hand-crafted downstream tasks, e.g., specific robotics tasks.…”

“…Figure 4: Box images recorded from the front and right views, respectively.We used a model trained on an image dataset shown in Figure4presented byChamzas et al (2021). The training D R f and test D E f datasets each consisted of 5000 images containing four boxes arranged in 12 possible configurations, referred to as classes, recorded from the front camera angle (Figure4left).…”

Augment-Connect-Explore: a Paradigm for Visual Action Planning with Data Scarcity

“…We perform a grid search for w ε in the interval [−0.65, −0.05] with step size 0.1. The LPM is a two-layer, 100 nodes MLP, while the Siamese network for the SM is a shallow two convolutional layer network with a latent space dimension 12 as in [17]. We train the Siamese network for 100 epochs and perform HDBSCAN [18] clustering in the latent space Z of the model.…”