Model-free vision-based shaping of deformable plastic materials

Abstract

We address the problem of shaping deformable plastic materials using non-prehensile actions. Shaping plastic objects is challenging, since they are difficult to model and to track visually. We study this problem, by using kinetic sand, a plastic toy material which mimics the physical properties of wet sand. Inspired by a pilot study where humans shape kinetic sand, we define two types of actions: pushing the material from the sides and tapping from above. The chosen actions are executed with a robotic arm using image-based visual servoing. From the current and desired view of the material, we define states based on visual features such as the outer contour shape and the pixel luminosity values. These are mapped to actions, which are repeated iteratively to reduce the image error until convergence is reached. For pushing, we propose three methods for mapping the visual state to an action. These include heuristic methods and a neural network, trained from human actions. We show that it is possible to obtain simple shapes with the kinetic sand, without explicitly modeling the material. Our approach is limited in the types of shapes it can achieve. A richer set of action types and multi-step reasoning is needed to achieve more sophisticated shapes. Figure 2. Human plastic material shaping study. Left: setup for the user study: an RGB-D camera (RealSense) is fixed above the sandbox, where participants mold the kinetic sand into a desired shape; first with one, then with two hands and third, using one of two provided tools. Center: four different shapes formed by the participants of our user study. Right: eight images (top: RGB, bottom: depth) acquired while a user is shaping the kinetic sand using both hands.

Deformable Elasto-Plastic Object Shaping using an Elastic Hand and Model-Based Reinforcement Learning

“…In terms of designing control or action selection policies to effectively manipulate deformable solid objects, prior work has utilized both model-based [18], [19] and modelfree adaptive methods [20], [4]. While simulation-based methods have been shown to be quite useful in the space of linear and planar object manipulation [7], [21], it is more computationally expensive to collect large amounts of data for physically-realistic simulations of dough.…”

“…While simulation-based methods have been shown to be quite useful in the space of linear and planar object manipulation [7], [21], it is more computationally expensive to collect large amounts of data for physically-realistic simulations of dough. Learning from demonstrations has therefore been cited to be the most popular method for training a robot to shape deformable objects [22], accomplishing tasks such as shaping sand [4] and pizza dough [23]. Similar to the work in [18], [19], this work uses a model-based method, where material-relevant parameters are estimated and then used to guide the control strategy.…”

“…Manipulation of clay or dough is typically non-prehensile, i.e., it involves pushing or moving the object against a surface [4]. In this work, we design a simple action space to roll out a piece of dough against a horizontal flat surface and track the dough's shape progression with a low-dimensional feature space.…”

Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review

“…They use a Jacobian-based method to drive the points within the deformable object toward a set of targets, assuming that they are able to sense the geometry of the object. Similar approaches are used in Cherubini et al (2020), Hirai et al (2001), and Wada et al (2001). These methods align points of interest on the deformable object to targets, using a visual-servoing controller or a proportionalintegral-derivative (PID) controller.…”

Vision-based manipulation of deformable and rigid objects using subspace projections of 2D contours

“…An expository paper on the topic is available in (Navarro-Alarcon et al, 2019). A recent work on vision-based shape servoing of plastic material was presented in (Cherubini et al, 2020).…”