Using structural bootstrapping for object substitution in robotic executions of human-like manipulation tasks

Abstract

In this work we address the problem of finding replacements of missing objects that are needed for the execution of human-like manipulation tasks. This is a usual problem that is easily solved by humans provided their natural knowledge to find object substitutions: using a knife as a screwdriver or a book as a cutting board. On the other hand, in robotic applications, objects required in the task should be included in advance in the problem definition. If any of these objects is missing from the scenario, the conventional approach is to manually redefine the problem according to the available objects in the scene. In this work we propose an automatic way of finding object substitutions for the execution of manipulation tasks. The approach uses a logic-based planner to generate a plan from a prototypical problem definition and searches for replacements in the scene when some of the objects involved in the plan are missing. This is done by means of a repository of objects and attributes with roles, which is used to identify the affordances of the unknown objects in the scene. Planning actions are grounded using a novel approach that encodes the semantic structure of manipulation actions. The system was evaluated in a KUKA arm platform for the task of preparing a salad with successful results.

Library of actions: Implementing a generic robot execution framework by using manipulation action semantics

“…parameters related to the kinematic chain or to the sensor hardware). As a consequence, the framework proposed here has already been used as an action execution routine in different robotic applications (Agostini et al, 2015; Wörgötter et al, 2015). In the work of Agostini et al (2015), we showed that the robot can still generate similar actions by replacing tools in manipulations using the aspect of tool affordance.…”

“…As a consequence, the framework proposed here has already been used as an action execution routine in different robotic applications (Agostini et al, 2015; Wörgötter et al, 2015). In the work of Agostini et al (2015), we showed that the robot can still generate similar actions by replacing tools in manipulations using the aspect of tool affordance. The work introduced in Wörgötter et al (2015) showed that robots can apply bootstrapping at different cognitive levels to improve their behavior based on the action representation and generation method proposed here.…”

Structural Bootstrapping—A Novel, Generative Mechanism for Faster and More Efficient Acquisition of Action-Knowledge

“…We have indeed now implemented a more complex scenario ("making a salad"), where structural bootstrapping happens on-line (during the execution of the task) providing additional support to the here presented concepts [47]. Both claims, however, will probably get more and more substantiated the richer the individual knowledge bases at the different layers will become in the future.…”

“…Still, a complete robotic implementation of these processes is currently being performed using the our robot systems [47]. For brevity, we will here show one central part of this implementation demonstrating the required transfer of human action knowledge [see Fig.…”

Learning how a tool affords by simulating 3D models from the web

“…As for outputs they can be scores for pixels/regions of an image (RGB, RGB-D) [6], [1] (in what is sometimes called pixel-wise labeling task); or for 'parts' of a point cloud [2], or a score for the object as a whole [9]. The outputs also vary in giving out a binary score (affords or does not afford) [12] or a graded score [1] [2], [11], [13], [14], or additionally providing manipulation cues that the robot could use to grasp [8], [1] and orient the tool. This final output (how the tool would be grasped oriented) is something we tackle in this paper, and is rarely considered in the existing literature; Mar et al [10] implicitly considers how the tool is grasped and oriented as part of its input, hence learning a function that accounts for how the affordance differs depending on how it is used.…”