SE3-Pose-Nets: Structured Deep Dynamics Models for Visuomotor Control

Abstract

In this work, we present an approach to deep visuomotor control using structured deep dynamics models. Our deep dynamics model, a variant of SE3-Nets, learns a low-dimensional pose embedding for visuomotor control via an encoder-decoder structure. Unlike prior work, our dynamics model is structured: given an input scene, our network explicitly learns to segment salient parts and predict their poseembedding along with their motion modeled as a change in the pose space due to the applied actions. We train our model using a pair of point clouds separated by an action and show that given supervision only in the form of point-wise data associations between the frames our network is able to learn a meaningful segmentation of the scene along with consistent poses. We further show that our model can be used for closedloop control directly in the learned low-dimensional pose space, where the actions are computed by minimizing error in the pose space using gradient-based methods, similar to traditional model-based control. We present results on controlling a Baxter robot from raw depth data in simulation and in the real world and compare against two baseline deep networks. Our method runs in real-time, achieves good prediction of scene dynamics and outperforms the baseline methods on multiple control runs. Video results can be found at: https://rse-lab.cs. washington.edu/se3-structured-deep-ctrl/

The limits and potentials of deep learning for robotics

“…Different variants of deep learning have been shown to successfully learn predictive physics models and robot control policies in a purely data-driven way (Agrawal et al, 2016; Byravan et al, 2018; Jonschkowski et al, 2017; Watter et al, 2015). Although such a learning-based paradigm could potentially inherit the robustness of intuitive physics reasoning, current approaches are nowhere near human prediction and control capabilities.…”

Self-Supervised Learning for Specified Latent Representation

“…Some studies use an autoencoder to learn a low-dimensional latent representation of input images and then perform specific tasks based on the latent representation. Byravan et al [2] design SE3-Pose-NETS, which learns a lowdimensional pose embedding for visuomotor control via an autoencoder structure. The low-dimensional pose is used as the input to a three-layer network that is used to predict the 6D pose.…”

Learning Latent Space Dynamics for Tactile Servoing

“…Our work is mostly inspired by previous works on learning control and dynamics in the latent space [9], [10]. Both of these works learn a latent space representation of the state, and also learn a dynamics model in the latent space.…”