A model-based approach to finding substitute tools in 3D vision data

Abstract

A robot can feasibly be given knowledge of a set of tools for manipulation activities (e.g. hammer, knife, spatula). If the robot then operates outside a closed environment it is likely to face situations where the tool it knows is not available, but alternative unknown tools are present. We tackle the problem of finding the best substitute tool based solely on 3D vision data. Our approach has simple hand-coded models of known tools in terms of superquadrics and relationships among them. Our system attempts to fit these models to point clouds of unknown tools, producing a numeric value for how good a fit is. This value can be used to rate candidate substitutes. We explicitly control how closely each part of a tool must match our model, under direction from parameters of a target task. We allow bottom-up information from segmentation to dictate the sizes that should be considered for various parts of the tool. These ideas allow for a flexible matching so that tools may be superficially quite different, but similar in the way that matters. We evaluate our system's ratings relative to other approaches and relative to human performance in the same task. This is an approach to knowledge transfer, via a suitable representation and reasoning engine, and we discuss how this could be extended to transfer in planning.

Learning how a tool affords by simulating 3D models from the web

“…Regarding representations they can use purely Machine Learning to learn features [8], [6], [1] or combine it with some hand-engineered features (e.g. histograms [2]; model fitting [9]). The grounding of the system concerns whether the approach is based on learning from hand-labelled data [2]; simulation of tasks [6], [10]; or a real robot trying out the tools [11].…”

“…The grounding of the system concerns whether the approach is based on learning from hand-labelled data [2]; simulation of tasks [6], [10]; or a real robot trying out the tools [11]. As for outputs they can be scores for pixels/regions of an image (RGB, RGB-D) [6], [1] (in what is sometimes called pixel-wise labeling task); or for 'parts' of a point cloud [2], or a score for the object as a whole [9]. The outputs also vary in giving out a binary score (affords or does not afford) [12] or a graded score [1] [2], [11], [13], [14], or additionally providing manipulation cues that the robot could use to grasp [8], [1] and orient the tool.…”

“…Nonetheless, we go beyond current approaches in our ability to output a graded score and both where to grasp and also how to orient the tool for a task, while considering both the visual and physical (weight) aspects of an object. Relative to our past work [9] the major difference is that we now learn a function relating tool parameters to affordance by simulating different ways of using tools that we gather from 3D Web models (whereas previously the function was hand coded).…”

“…Superquadrics have been used in Computer Vision for their ability to represent many different shapes and recovery of superquadrics from range data has been widely studied [15]. We fit superquadrics as in our previous work [9] and extend the idea to also recover superparaboloids. We refer the reader to [15] as the main source of our approach to superquadrics.…”

“…The parameters a 1 , a 2 and a 3 control the scale and 1 the curvature. Each superquadric/paraboloid takes 13 parameters to represent: 5 for scale and shape, 2 for tapering 6 more for general orientation and position in space [9].…”