Perceiving, learning, and exploiting object affordances for autonomous pile manipulation

Abstract

Abstract-Autonomous manipulation in unstructured environments presents roboticists with three fundamental challenges: object segmentation, action selection, and motion generation. These challenges become more pronounced when unknown manmade or natural objects are cluttered together in a pile. We present an end-to-end approach to the problem of manipulating unknown objects in a pile, with the objective of removing all objects from the pile and placing them into a bin. Our robot perceives the environment with an RGB-D sensor, segments the pile into objects using non-parametric surface models, computes the affordances of each object, and selects the best affordance and its associated action to execute. Then, our robot instantiates the proper compliant motion primitive to safely execute the desired action. For efficient and reliable action selection, we developed a framework for supervised learning of manipulation expertise. We conducted dozens of trials and report on several hours of experiments involving more than 1500 interactions. The results show that our learning-based approach for pile manipulation outperforms a common sense heuristic as well as a random strategy, and is on par with human action selection.

Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction

“…Aforementioned online methods do not use learning to generalize to new object scenes, whereas our approach learns a visual model that improves segmentation for new object scenes using data gathered from over 2,300 robot-object interactions. Katz et al [15] present an interactive segmentation algorithm based on a learned model for detecting favourable actions to remove object clutter. An interesting recent overview on interactive perception is presented by Bohg et al [7], who summarize that perception is facilitated by interaction with the environment.…”

Model-Based Grasping of Unknown Objects from a Random Pile

“…Hence, the grasps are learned from images and this method can be classified as an empirical approach. In Reference [19], the authors use a learning-based approach and 1500 interactions for learning to clear out a pile of objects. The unknown objects are segmented into hypothesised object facets.…”

Open source hardware and software platform for robotics and artificial intelligence applications

“…Image processing plays an important role in robotics applications [8]. There are still many unsolved problems in a simple set up such as controlling a robot to manipulate objects [9].…”