Learning haptic representation for manipulating deformable food objects

Abstract

Abstract-Manipulation of complex deformable semi-solids such as food objects is an important skill for personal robots to have. In this work, our goal is to model and learn the physical properties of such objects. We design actions involving use of tools such as forks and knives that obtain haptic data containing information about the physical properties of the object. We then design appropriate features and use supervised learning to map these features to certain physical properties (hardness, plasticity, elasticity, tensile strength, brittleness, adhesiveness). Additionally, we present a method to compactly represent the robot's beliefs about the object's properties using a generative model, which we use to plan appropriate manipulation actions. We extensively evaluate our approach on a dataset including haptic data from 12 categories of food (including categories not seen before by the robot) obtained in 941 experiments. Our robot prepared a salad during 60 sequential robotic experiments where it made a mistake in only 4 instances.

Explorative Experiments in Autonomous Robotics

“…In total, we consider 11 papers that we deem represent a significant sample of current research on autonomous robots, as they are witnessing the awarded research in two of the main conferences of the field (see Table 1). [Hoffman and Weinberg, 2010] Gesture-based human-robot jazz improvisation [Grollman and Billard, 2011] Donut as I do: Learning from failed demonstrations [Bergstrom et al, 2011] Generating object hypotheses in natural scenes through human-robot interaction [Thobbi et al, 2011] Using human motion estimation for human-robot cooperative manipulation [Tenorth et al, 2012] The RoboEarth language: Representing and exchanging knowledge about actions, objects, and environments [Daniel et al, 2012] Learning concurrent motor skills in versatile solution spaces [Chu et al, 2013] Using robotic exploratory procedures to learn the meaning of haptic adjectives [Fasola and Mataric, 2013] Using semantic fields to model dynamic spatial relations in a robot architecture for natural language instruction of service robots [Deisenroth et al, 2014] Multi-task policy search for robotics [Gemici and Saxena, 2014] Learning haptic representation for manipulating deformable food objects [Boularias et al, 2015] Grounding spatial relations for outdoor robot navigation…”

“…[Grollman and Billard, 2011] "Because there are more possibilities to explore, in our experiments the donut method took more interactions to succeed than the balanced mean" (p. 3808) [Bergstrom et al, 2011] "Again, we conclude that point initialization outperforms cluster initialization" (p. 832) "In addition we evaluate how the method in [10] compares to our method" (p. 833) [Thobbi et al, 2011] " Fig. 8 shows the trajectories of both ends of the table for cases where the proposed system was used (case I: with predictions) and the case where only the reactive controller was used (case II: without predictions)" (p. 2877) [Daniel et al, 2012] "We also compare our approach to the standard unimodal REPS algorithm" (p. 3595) [Gemici and Saxena, 2014] "We compare the performance of our reward based manipulation approach against the baseline algorithms" (p. 644) Controlled experiments. These experiments are the golden standard of experimentation in the natural sciences that refers to the original idea of experiment as controlled experience, where the activity of rigorously controlling (by adopting experimental principles such as reproducibility or repeatability) the factors that are under investigation is central, while eliminating the confounding factors, and allowing for generalization and prediction.…”

“…[…] In a second experiment, we evaluate the influence of importance sampling" (p. 3595) [Gemici and Saxena, 2014] "In this work, we manually tuned the reward functions for our manipulation task for a reasonable level of exploration and exploitation" (p. 644) Confirmation of expectations or hypotheses. When developing robot systems, the designers consider (and build upon) a set of expectations and hypotheses about the behavior of the artifacts when inserted in their operating environments.…”

“…5 shows the variation confidence (C) through the task" (p. 2877) "the trajectory is much smoother when the human is placing the table down as compared to moving upwards […] It can also be speculated that sophisticated velocity or torque controlled robots would yield smoother motions and offer better improvements in performance using the proposed technique" (p. 2878) [Fasola and Mataric, 2013] "To illustrate the usefulness of the semantic field model towards representing static and dynamic spatial relation primitives for use in path generation and classification, Fig. 6 shows the progression of the at, along, away from, and in semantic field values along the execution paths generated for test runs #1-4, respectively" (p. 148) [Gemici and Saxena, 2014] "This means that for most of the objects, one or two information gathering actions was enough to determine the best task oriented action to reach the subgoal" (p. 644) [Boularias et al, 2015] "We notice that complex commands help finding the right goals because they are less ambiguous than simple commands" (p. 1981) Assessing the generality of robot systems. The most sophisticate way of employing experiments with an explorative flavor is to gain knowledge about the behavior of robot systems in settings that are different of those considered in their design, in order to evaluate the generality of these systems.…”

Estimating the deformability of elastic materials using optical flow and position-based dynamics

“…The predominant methodology is based on haptic sensors [23], [24], [25], [26]. An interesting work [27], proposes the design of feature descriptors to capture the properties of semi-solid objects and to recognize objects from haptic observation in a supervised manner. In addition, some work has been done on extracting such properties using visual perception alone [28], [29].…”