Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation

Abstract

Haptic feedback is essential for humans to successfully perform complex and delicate manipulation tasks. A recent rise in tactile sensors has enabled robots to leverage the sense of touch and expand their capability drastically. However, many tasks still need human intervention/guidance. For this reason, we present a teleoperation framework designed to provide haptic feedback to human operators based on the data from camera-based tactile sensors mounted on the robot gripper. Partial autonomy is introduced to prevent slippage of grasped objects during task execution. Notably, we rely exclusively on low-cost off-the-shelf hardware to realize an affordable solution. We demonstrate the versatility of the framework on nine different objects ranging from rigid to soft and fragile ones, using three different operators on real hardware.

Learning Task Constraints in Visual-Action Planning from Demonstrations

“…Instead an important work in this regard but in the field of control is Embed to Control [6], where the authors build on insights from the optimal control formulation to leverage a variational autoencoder (VAE), that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. The work in [1] also uses VAEs but with augmented loss function to generate Visual Action Plans making use of a Latent Space Roadmap that produces a sequence of images as well as connecting actions given only a start and goal image. However, since these approaches do not operate in a semantically-interpretable latent space, it is difficult to determine whether plans or sequences of actions are satisfying a given set of specifications.…”

“…We propose an architecture that learns to distinguish between system executions that satisfy the user-defined requirements and undesired executions from provided demonstrations. The architecture extends the visual action planning framework in [1] by integrating binary classifiers to evaluate the satisfaction of the constraints without sacrificing the benefits of data-driven low-dimensional latent space representations.…”

“…To enable designers to easily specify task constraints in visual planning frameworks, this work 1) learns task constraints from demonstrations using LSTMs; 2) demonstrates the accuracy of the constraint evaluation in the latent space representation; 3) uses the structure of linear temporal logic to compose complex constraints; and 4) extends the visual-action planning framework in [1] to allow jumps between different latent states so that humans can interact with the system, e.g., by removing boxes; The remainder of the paper is organized as follows: Sec. II provides necessary preliminaries.…”

“…Subsequently, Sec. III presents how to learn constraints and their integration in the visual-action planning framework presented in [1]. In Sec.…”

“…The recent advances in representation learning have enabled novel planning approaches, such as visual planning [1], which require significantly less training effort. These approaches allow one to automatically create low dimensional latent space representations that can be used for planning without explicitly modelling the state space specifically for the task.…”